Building a Time-Tracking SaaS: Lessons From the Field

I keep coming back to one specific moment from this project when I try to explain why I care so much about correctness over raw performance in backend work: a support ticket, maybe four months into building this thing, from a field supervisor who said one of her team members had been marked as clocked out for an hour in the middle of a shift that he swore he worked the whole time, standing right there with his phone in his pocket the entire shift. It wasn't a dramatic bug. Nothing crashed. No error appeared anywhere. The system just quietly recorded the wrong thing, confidently, and a real person's pay was about to be wrong for it. That ticket taught me more about what this kind of system actually demands than any architecture diagram ever could have.

The product itself was a workforce time-tracking SaaS — I'll describe it at that level without naming it, since the specifics of the brand matter less than the shape of the problem. Field teams checked in and out of shifts using a selfie-based verification step, their location got tracked continuously through the shift for geofencing purposes, and all of that fed into timesheets that, eventually, became payroll. Which means the system had exactly one job it absolutely could not get wrong, even occasionally, even rarely: knowing, accurately, when someone worked and where they were while they did it. Everything else about the product — dashboards, reports, admin tooling — could have a rough day and nobody would really suffer for it. The timesheet accuracy could not have a rough day, ever, because a rough day there became someone's paycheck being wrong, and people notice that immediately, and they should.

The stack, and why the hardest problem wasn't a feature

The backend ran on Laravel, with Redis handling the hot, fast-moving state — live tracking positions, active-shift session data, the stuff that needed to be read and written constantly while a shift was in progress — and Docker keeping development, staging, and production environments consistent enough that "works on my machine" wasn't a phrase that got to excuse anything. None of that is unusual technology for this kind of product. What made the project genuinely hard wasn't any single feature on the list — not the selfie verification, not the geofencing logic, not the reporting dashboards. It was something that doesn't show up on a feature list at all: making sure that geofencing, selfie validation, and raw GPS location pings could all write to the same timesheet record, arriving in whatever order a flaky mobile network happened to deliver them, without one update accidentally clobbering a more correct, more recent one.

To understand why that's hard, you have to understand how genuinely unreliable mobile connectivity is for field workers, in a way that's easy to forget if you mostly build software that assumes a stable connection. A worker's phone doesn't send one clean, orderly stream of location updates. It buffers several location pings locally when the signal drops — inside a building with bad reception, in a rural area, in a basement — and then, the moment it reconnects, it fires off that entire backlog of buffered pings in a rapid burst. Those pings carry timestamps from whenever they were actually recorded, not from when they finally arrived at the server. So the server might receive, within the same second, a ping timestamped two minutes ago, one timestamped forty seconds ago, and one timestamped right now, completely out of chronological order relative to the order they arrived in. If the backend just took "whatever arrived most recently" as truth, without checking what time that ping actually represented, you get exactly the kind of bug that ticket described: a late-arriving, buffered ping from a moment when the worker's connection had already dropped briefly overwrites a correct, more current position with stale information, and the system now believes something false about where that person was and whether they were inside the required work zone.

Living inside the race condition

I want to walk through what actually happens, moment by moment, in that failure mode, because the first time I understood it fully I felt a specific kind of dread — the feeling of realizing a bug isn't rare at all, it's happening constantly, you just hadn't noticed the pattern yet. Picture a worker whose phone loses signal walking into a warehouse. Their location app keeps recording where they are internally, buffering those readings locally since it can't reach the server. Three minutes later they walk back outside, signal returns, and the buffered readings all get sent at once, arriving within a second or two of each other. Meanwhile, a different part of the same worker's phone — or in some cases a completely fresh, live reading taken the instant connectivity returned — also sends an up-to-the-second ping. If the server processes that fresh, current ping first, and then immediately afterward processes one of the three-minutes-old buffered pings because it happened to arrive a beat later in the queue, the system now overwrites the correct, current position with a stale one from three minutes ago. The timesheet record for that shift now says, falsely, that this person was somewhere they weren't, at a time they were actually somewhere else — which is exactly the kind of thing that can make a geofence check fail incorrectly and mark someone as having left the job site when they never did.

The fix, once I'd actually sat with the problem long enough to see its real shape, turned out to be conceptually simple, even though getting it reliable under real concurrency took some care. Every time a location update came in for a given worker, the system had to do two things before touching that worker's timesheet record at all. First, it had to make sure that no other update for that exact same worker was being processed at the exact same moment — because two updates landing within milliseconds of each other, both trying to read-then-write the same record, is its own way of corrupting data even without the out-of-order timestamp problem, through what's usually called a race condition: two processes each assume they have the latest picture of a record, both make a decision based on that assumption, and whichever one finishes writing last wins, regardless of which one was actually correct. So I used a short-lived lock, scoped specifically to that one worker, that any update for them had to acquire before doing anything else — meaning if two pings for the same person arrived close together, the second one would simply wait its very brief turn rather than racing the first.

Second, and just as important, once an update had its exclusive turn, it had to check: is the timestamp on this update actually newer than the timestamp already stored on this worker's record? If not — if this is one of those buffered, stale pings arriving late — the system discards it entirely rather than writing it. That one check, an explicit comparison against whatever was already recorded, is what actually prevented the overwrite. Without it, the lock alone wouldn't have helped at all, because the lock only prevents two updates from stepping on each other simultaneously; it says nothing about which of two sequential updates represents a more accurate picture of reality. You need both pieces together: exclusivity, so updates don't corrupt each other mid-write, and ordering awareness, so an update actually knows whether it's allowed to speak for "now" or whether it's really speaking for an unreliable "a few minutes ago" that's already been superseded.

Finding it, which was its own kind of work

I didn't arrive at that explanation cleanly or quickly. The ticket that kicked this off described a symptom — a worker marked as clocked out when he wasn't — but symptoms like that don't point directly at their cause. My first instinct, which in hindsight was wrong but a reasonable place to start, was to suspect the geofencing radius itself, wondering if the zone was drawn too tightly and a normal GPS wobble near the boundary was tripping a false exit. I spent close to a full day pulling location history for that specific worker's shift and plotting it, only to find his positions were well inside the geofence the entire time — which ruled out the geometry and told me the problem had to be in how positions were being recorded in the first place, not in how they were being evaluated against a boundary.

That's when I started looking at raw timestamps on the stored pings rather than just the positions, and noticed something that should have been obvious sooner: the timestamp on the ping that triggered the false "left the zone" event was earlier than the timestamp on an update that had already been successfully recorded for that same worker a few minutes before. In other words, an older update had landed after a newer one and silently won. Once I saw that one case clearly, I went looking for how common it actually was across other shifts and other workers, rather than assuming it was a one-off. It wasn't rare at all — once I knew the signature to search for, out-of-order pings turned out to be happening dozens of times a day across the active workforce, almost all of them harmless because the stale and current positions usually happened to be close enough together that no visible symptom resulted. The supervisor's ticket wasn't a freak occurrence. It was the one time, out of many quiet near-misses, where the stale and current positions happened to differ enough, and happened to straddle a geofence boundary, to actually produce a wrong, visible, consequential result. That realization — that the bug had probably been happening invisibly for a while before anyone noticed it loudly enough to report — was uncomfortable in a way that stuck with me well past fixing it.

The companion tool, and a different kind of reliability problem

Alongside the mobile and backend work, I also built a companion browser extension for the desktop side of the product, aimed at office-based staff who needed to review and adjust timesheet data without necessarily having the mobile app running. That piece used Vue.js for the interface and leaned on the browser's own local storage capabilities to keep a cached, usable copy of timesheet data on the user's machine, specifically so that someone working from an office with its own unreliable internet connection — which, perhaps ironically for an app meant to track field workers with bad connectivity, turned out to be more common among the desk staff than I expected — wouldn't be completely blocked from doing their job the moment their connection blipped. Changes made while offline queued locally and synced back to the server the moment connectivity returned, rather than the interface simply failing or showing a spinner indefinitely. It was a smaller, quieter problem than the race condition, but it rhymed with it in an interesting way: both were fundamentally about designing for a world where you cannot assume a clean, ordered, always-available network, and building software that degrades gracefully instead of just breaking when that assumption fails, which it always eventually does.

The disagreement with Tony

There was a colleague on this project, a product-minded engineer with the kind of swagger that always reminded me of Tony Stark walking into a room already certain he's got the answer, who pushed back hard on the lock-based fix when I first proposed it. His argument was that acquiring a lock for every single incoming location ping would add latency to a path that was already under pressure, and that we should instead just take the newest-arriving ping as truth and move on, trusting that out-of-order deliveries were rare enough not to matter in practice. It was a reasonable argument on the surface, and honestly, a faster one to ship. I didn't have hard data to counter him with in that first conversation, just the ticket, a single incident, which on its own is thin evidence to justify adding overhead to a hot path.

So I did the thing that actually settled it, instead of continuing to argue from intuition: I pulled a week's worth of raw location ping logs and wrote a one-off analysis script, outside the production codebase, that simply counted how often a ping for a given worker arrived with a timestamp earlier than the timestamp already stored for that worker at the moment of arrival. The number came back far higher than either of us expected, dozens of instances per day across the active workforce, and once I showed Tony the breakdown, sorted by how large the time gap was between the stale ping and the current reading, he went quiet for a moment and then said something like, "okay, yeah, I was wrong, do the lock." I appreciated that about him, actually — he argued his position with real conviction, but he didn't dig in once the data clearly didn't support it. We ended up implementing the lock together, actually, splitting the work between the acquisition logic and the ordering check, and testing it against a replayed version of that same week of logs to confirm it would have caught every one of those stale-overwrite cases if it had been running at the time. Watching the replay confirm the fix, case by case, against real historical data rather than a synthetic test scenario, was one of the more satisfying moments of the whole project.

Why selfie verification turned out to be its own quiet minefield

The selfie-based check-in step seems, from the outside, like a relatively contained feature: a worker opens the app, takes a photo of themselves, the system checks it against a stored reference photo, and either confirms the check-in or flags it for review. In practice it turned into a source of some of the most emotionally charged support conversations on the whole product, because a false rejection there doesn't just produce a bug report, it produces a worker standing outside a job site, unable to clock in, worried about being marked late or absent for something entirely outside their control. Lighting conditions at outdoor job sites varied enormously depending on time of day and weather. A worker starting a dawn shift in low light, or finishing one at dusk with the sun directly behind them, would sometimes get rejected by checks that worked perfectly fine for the exact same person photographed in an office at midday during testing.

I ended up spending real time tuning the sensitivity of that matching process specifically for these edge lighting conditions, which meant collecting a deliberately messy, realistic set of sample photos taken under actual field conditions rather than relying on clean, well-lit test photos the way the feature had originally been validated. It also meant building in a graceful fallback: if the automated check came back uncertain rather than confidently right or wrong, the system queued that check-in for a quick manual review by a supervisor instead of outright rejecting it, so a worker in bad lighting wasn't punished for something that had nothing to do with whether they were actually who they claimed to be. That fallback path probably prevented more frustration, and more eroded trust in the product, than any amount of further tuning the core matching sensitivity could have, because it acknowledged honestly that the automated check would never be perfect and built a humane path around its imperfection rather than pretending the imperfection didn't exist.

The geofencing boundary problem that wasn't really about geofencing

Separately from the race condition, there was a recurring complaint early on about geofence boundaries feeling "too strict," with workers occasionally flagged as having left a job site when, by their own account, they never moved. I initially treated this as a tuning problem — maybe the configured radius around a site was simply drawn too tight for how GPS accuracy actually behaves in the real world, where a phone's reported position can drift by several meters even while the person holding it stands perfectly still, especially near tall buildings or dense tree cover that interferes with satellite signal.

That turned out to be a real contributing factor, but not the whole story, and untangling the two issues from each other took longer than I expected because they produced similar-looking symptoms. Some of the "left the geofence" false alarms were genuinely about GPS drift at the boundary, and those got meaningfully better once I introduced a small buffer zone and a brief dwell-time requirement before actually flagging an exit — meaning a single wobbly reading just outside the line wouldn't immediately trigger an alert; the system would wait a short window to see if the position settled back inside before treating it as a real departure. But a separate subset of those same complaints turned out to be the stale-ping race condition wearing a geofencing costume: an old, buffered position from genuinely outside the zone overwriting a current, accurate position that was genuinely inside it, exactly the failure mode from that original ticket, just reported by a different worker in a different city. Recognizing that two visually identical symptoms had two completely different root causes, one about GPS physics and one about data ordering, was its own small lesson in not assuming a bug has a single explanation just because its symptom looks consistent across reports.

The texture of maintaining this under growth

As the customer base grew, the character of the work shifted from "find and fix the sharp, surprising bugs" to something steadier and less dramatic: watching dashboards, noticing gradual upward trends before they became incidents, and making small proactive adjustments. I started treating certain metrics almost like a pulse I checked daily out of habit rather than because anything was actively wrong — average location-update latency during peak hours, the rate of locks timing out under contention, the gap between when a shift's data landed in Redis and when it got reliably persisted into MySQL for longer-term storage and reporting. Catching a slow upward creep in lock contention weeks before it would have become a user-visible slowdown, just by watching that one unglamorous metric regularly, felt less exciting than fixing the original race condition but was, in its own quiet way, just as valuable, because it meant customers never experienced the problem at all rather than experiencing it and then watching us fix it.

When the hot path started to strain

The locking-and-ordering fix solved the correctness problem, but it didn't solve a second problem that showed up later, once the product had more customers and more workers actively on shift at the same time: the Redis-backed hot path, the one handling all this live tracking state, started to strain once concurrently active shifts climbed into the low thousands. Latency on location updates crept up during peak hours — mid-morning and early afternoon, when the overwhelming majority of shifts across all customers happened to be active simultaneously — to the point where supervisors watching live maps started noticing a lag between a worker's actual position and what the dashboard showed.

Fixing that took a different kind of attention than the race condition had, less about a single sharp insight and more about a series of smaller, additive improvemements. Connection pooling cut down on the overhead of constantly opening and tearing down connections to Redis under heavy concurrent load. Batching multiple geofence checks into a single round trip to Redis, instead of issuing one separate round trip per individual location ping, cut the raw number of network calls dramatically during peak load, since most of that latency wasn't computation time at all, it was simply the cumulative cost of too many small, separate conversations happening between the application and Redis every second. And probably the highest-leverage change of the three: recognizing that not everything needed to live on the fast, expensive hot path at all. Daily summaries, weekly reports, anything that didn't need to reflect reality within the last second or two, got pulled off Redis entirely and rebuilt as scheduled aggregation jobs that read from MySQL on their own slower cadence. That single decision — being honest about which data genuinely needed sub-second freshness and which data was being kept hot purely out of habit rather than actual need — did more to relieve pressure on the hot path than any amount of further tuning within Redis itself could have.

What it felt like to carry this

I think what made this project different from a lot of backend work I'd done before it was the specific, quiet weight of knowing that a bug here wasn't an inconvenience, it was someone's income. That changes how carefully you move. I found myself, more than once, rewriting a piece of logic I'd already shipped and already tested, not because a test had failed, but because I'd thought about some edge case in the shower or on a walk and couldn't shake the feeling that it wasn't actually covered. I don't think that's an efficient way to build software in general, and I wouldn't recommend being perpetually anxious as a development methodology. But for this particular system, that low hum of "what haven't I thought of yet" turned out to be exactly the right instinct to listen to, because the bug that actually surfaced — the one from that first ticket — was precisely the kind of thing a normal test suite, built around clean, well-ordered inputs, would never have caught on its own. Real users don't send clean, well-ordered inputs. They send whatever their phone's radio happened to manage, in whatever order a flaky tower handed it back.

A second strain on the hot path, further out

Months after the initial Redis tuning had settled things down, a second, subtler strain showed up, this time not from raw volume but from a change in usage pattern: a few large customers started running overlapping shift schedules across bigger teams, which meant the distribution of when workers were actively tracked shifted from a single predictable midday peak to several smaller peaks spread across more of the day, including hours we hadn't originally tuned for. Our existing scaling assumptions, built around one dominant peak window, didn't quite fit anymore, and for a short stretch we saw latency creep back in during one of those newer, secondary peak windows we hadn't been watching as closely. Catching that meant going back to the same unglamorous dashboards I'd started treating as a daily habit, noticing the new pattern, and adjusting the scheduled aggregation jobs and connection pool sizing to account for a flatter, more spread-out load profile rather than the single-peak assumption everything had originally been tuned around. It was a good reminder that tuning a system once, even carefully, isn't a permanent state; the shape of real usage keeps shifting underneath you as the customer base changes, and the discipline has to be ongoing rather than a box you check once and move past.

How it settled

By the time I moved off active development on this part of the product, the lock-and-ordering fix had been running in production for months without a single recurrence of that original symptom, across a workforce that had grown considerably since the ticket that started this whole investigation. The hot-path tuning held up through subsequent growth too, handling peak concurrent shifts well beyond what had originally caused the slowdown, without needing another dramatic intervention — just the occasional small adjustment as load patterns shifted. What I keep from this project isn't really a specific technique, even though the lock-plus-ordering-check pattern is one I've reused since in other contexts involving unreliable, out-of-order input. It's more a disposition: that in a system where correctness failures are invisible until they aren't — until a supervisor files a ticket, until someone's paycheck looks wrong — the actual engineering work is in hunting down the quiet cases that don't announce themselves, not in making the obvious, visible cases faster. The visible cases usually take care of themselves. It's the silent ones that cost people money and trust, and those are the ones worth losing a little sleep over.