What I Learned Leading MIS Platforms for 50M+ Users

There was a stretch of about two years where my phone would buzz at six in the morning, India time, and I'd already know before looking at it: a dashboard had gone red overnight, and somewhere in a state office a program director was going to ask why the numbers didn't match what the field staff were reporting. That was the rhythm of working on large-scale MIS platforms for government programs — systems meant to track the activity of field agents, health workers, and enumerators spread across a country's worth of districts, villages, and blocks, feeding data up through a pipeline that was supposed to end in a clean number a bureaucrat could trust enough to make a decision from.

I want to tell this story carefully, because the specific programs and the specific agencies I worked with aren't something I'm going to name here — partly for confidentiality reasons that predate this write-up, and partly because the lesson has nothing to do with which program it was. The lesson is about what happens when you try to build a system of record out of data that arrives from thousands of people you've never met, using devices you didn't choose, on networks you have no control over. That's a different kind of engineering problem than almost anything else I've worked on, and it took me longer than I'd like to admit to understand why.

How I Ended Up There

I didn't walk into this job thinking I'd become the person who cared about GPS radius checks and EXIF metadata. I came in as a backend engineer who was reasonably good with Laravel, comfortable with queues and scheduled jobs, and had built a handful of admin dashboards before. The brief, as it was explained to me in the first week, sounded almost boring on paper: build the reporting layer for a system that already had field data collection apps in place, aggregate it, and expose it through dashboards for program coordinators at the district, state, and national level.

It took about three weeks of actually looking at the data for that boring brief to turn into something that kept me up at night. The first thing I noticed, going through a sample export someone handed me to get oriented, was that the submission timestamps and the submission locations didn't line up the way you'd expect. An agent would be logged as submitting a home visit record in one village at 9:14 in the morning, and then another record eleven minutes later from a village that was, by road, forty minutes away. Either this agent had access to a helicopter, or something in the pipeline was letting through data that nobody had actually looked at critically.

I raised it in a meeting, expecting someone to tell me it was a known issue, already handled somewhere downstream. Instead I got a slightly uncomfortable silence, and then an admission: nobody had built anything to catch that. The existing system took whatever the mobile app sent, wrote it to the database, and that was it. Validation, to the extent it existed, happened weeks later, when someone in a state office manually cross-checked a sample of records against paper registers — a process that caught maybe one percent of problems and caught them months after the damage of a bad decision had already been made from bad numbers.

That was the moment the job stopped being "build dashboards" and became "figure out what it even means for this data to be trustworthy."

What Thousands Of Field Agents On Bad Networks Actually Means

It's worth spelling out what the operating conditions actually looked like, because I think it's easy to underestimate just how hostile the environment was to the kind of clean, predictable data flow you'd design for in a comfortable urban deployment. We're talking about field agents — community health workers, registration staff, outreach coordinators — working across rural and semi-urban areas with patchy two-g and three-g coverage, carrying low-end Android devices that the program had procured in bulk, often the cheapest models that could still run the required app. Power cuts were routine. A lot of agents worked in areas where you'd lose signal mid-sentence on a phone call, and the app had to assume that every single submission might sit on a device for hours or days before it had enough signal to sync.

That single assumption — that data might be old by the time it arrived — turned out to ripple through almost every design decision that followed. You couldn't trust "submitted at" to mean "happened at." You couldn't assume network calls would succeed on the first try, or the fifth. You couldn't assume the device's clock was even set correctly, because I found more than one agent's device whose system clock had drifted by days, presumably never corrected because the agent had no reason to notice or fix it. Every one of those small, unglamorous facts about the real world had to become an explicit decision in the system, because the alternative was pretending the world was tidier than it was and getting burned later.

And there were a lot of agents. Tens of thousands, submitting records throughout the day, with submission volume that spiked around specific reporting deadlines — the last two days of a month would see submission volume triple compared to a quiet Tuesday mid-month, because agents who'd been putting off their paperwork would suddenly remember they had a quota to hit. That spiky, bursty pattern mattered enormously for how I ended up designing the ingestion and aggregation pipeline, something I'll get into shortly, because a system that works fine at steady-state can fall over completely during the three days a month when everyone needs it to work.

The First Real Failure: Trusting The Data As It Arrived

Before I describe what we built, I want to be honest about what didn't work, because that's where the real learning happened. My first instinct, when I understood the scale of the data quality problem, was to build a cleanup layer — a nightly job that would scan the previous day's submissions and flag the suspicious ones for a human to review. It felt like the responsible, low-risk approach: don't touch the live submission flow, don't risk breaking anything agents depended on, just add a safety net after the fact.

I built it. It worked, in the narrow sense that it ran every night and produced a list of flagged records. But it solved almost nothing, and it took me embarrassingly long to understand why. The flagged records piled up faster than anyone could review them — by the second week there were over four thousand flagged submissions sitting in a queue, and the two people whose job it nominally was to review them had other full-time responsibilities and looked at maybe thirty a day if they were diligent. More importantly, by the time a bad record got flagged, it had often already been counted in a weekly summary that had already gone up to a state coordinator, who had already made a decision or a public statement based on it. Catching a problem after it's been baked into a report that's already been acted on isn't catching it; it's documenting that a mistake happened.

I remember the specific conversation that made this click for me. A state-level coordinator asked, in a review call, why the submission count for one district had jumped by nearly thirty percent in a single week with no explanation. I went looking and found that a handful of agents had been submitting the same beneficiary record multiple times — not maliciously, as far as I could tell, but because the app's "submit" button didn't give clear enough feedback about whether a submission had gone through on a flaky connection, so an agent who wasn't sure it worked would just tap it again. Multiply that uncertainty by thousands of agents on unreliable networks, several times a day, and you get a meaningful chunk of your "growth" being an artifact of bad UX feedback loops rather than real program activity. That bad number had already been quoted in a meeting before anyone caught it.

That was the turning point. I went back to the team and argued, more forcefully than I usually do in those settings, that we needed to stop treating validation as a downstream cleanup step and start treating it as a gate — something a submission had to pass before it was allowed to exist as "data" at all, not something we fixed up afterward with apologies.

Building Validation At The Point Of Entry

Here's the part I think about the most, because it's the part that genuinely surprised me with how much nuance there was once I actually dug into it. The idea of "validate the submission before accepting it" sounds simple when you say it out loud. Figuring out exactly what to check, and how strict to be, took weeks of iteration, a lot of arguments with the program side about what was a legitimate edge case versus what was fraud, and more than one case where my first version of a rule turned out to reject perfectly honest agents doing their job under difficult conditions.

The first check I built was a location plausibility check: did the GPS coordinates attached to a submission make sense given which geographic zone that agent was assigned to. The naive version of this — just drawing a fixed radius around the zone's centroid and rejecting anything outside it — broke almost immediately in testing, because some of these assigned zones were enormous and irregularly shaped, stretching across terrain where a straight-line radius from the center point would reject legitimate submissions from the far edge of a zone while happily accepting a fabricated submission near the center. I ended up building the check against the actual boundary polygon of the assigned zone instead of a radius, with a tolerance buffer added around the edge to account for ordinary GPS drift — phone GPS chips in the low-end devices we were dealing with could be off by anywhere from twenty to a hundred and fifty meters depending on terrain and cloud cover, and I had to learn that the hard way after rejecting a batch of entirely legitimate submissions from agents working near a zone boundary and getting an angry phone call about it.

Even after fixing that, there was a subtler problem: what do you do about an agent whose device GPS straight up failed to get a fix at all, which happened more often than I expected, especially in areas with dense tree cover or inside concrete buildings? I had to decide, and this was genuinely a judgment call rather than something with an obviously correct answer, whether a missing or zero-value GPS reading should block the submission outright or just get flagged with lower confidence and let through with a warning. We ended up letting it through but marking it with a distinct status, because rejecting it outright meant an agent who'd done the actual work — visited the household, had the real conversation — would be penalized for a hardware limitation entirely outside their control, and that felt like exactly the wrong kind of system to build. The whole point was to catch dishonest or careless data, not to punish people for having a cheap phone in a bad signal area.

The second check was duplicate detection — catching the same agent submitting what was effectively the same record twice within a short window, the pattern that had caused that embarrassing thirty-percent spike. This one had its own wrinkle: how do you define "the same record" when the underlying data has small, honest variations between two submissions of what is genuinely the same event? An agent might submit a visit, not be sure it went through, resubmit it ten minutes later with the beneficiary's name typed slightly differently because autocomplete suggested something different the second time, or with a slightly different timestamp because the device clock ticked forward. I ended up matching on a combination of the agent identifier, the beneficiary identifier, and a time window rather than doing an exact match on every field, because an exact match was both too strict, in that it let through near-duplicates with trivial differences, and in another sense too permissive, because requiring every field to match exactly meant a slightly fat-fingered resubmission would sail through as "different."

The time window itself needed tuning too. I started with thirty minutes, which caught the obvious cases, but then a program officer pointed out a real workflow I hadn't accounted for: sometimes an agent genuinely does need to submit an updated or corrected version of a record they filed an hour or two earlier, after the beneficiary provided additional information in a follow-up conversation that same day. A duplicate-detection window that was too wide would start rejecting legitimate corrections, which would train agents to find workarounds — maybe by changing some unrelated field slightly just to dodge the duplicate check, which would be worse than the problem we were trying to solve, because now we'd be training the exact evasive behavior that hurts data integrity in a more deliberate way. We settled on sixty minutes for the strict duplicate flag, with anything submitted after that treated as a distinct event that would get surfaced to a supervisor as a potential correction rather than silently merged or silently rejected.

The third check, and the one I was initially most skeptical would work reliably, involved the photo evidence agents were required to attach to certain kinds of visits. The requirement existed because earlier, less digital versions of these programs had well-documented problems with agents fabricating visit records entirely — claiming to have visited a household when they hadn't. A photo requirement sounds like it solves that, until you realize a photo on its own proves almost nothing: anyone can take a photo of anything, at any time, and attach it to any record. What actually mattered was whether the photo's embedded metadata — the geotag information that most phone cameras write into the image file at the moment of capture — was present and consistent with the claimed location and roughly consistent with the claimed time. If an agent submitted a photo with no geo-metadata at all, that was a strong signal the photo had been pulled from the phone's existing gallery rather than taken fresh at the location, and we flagged those for mandatory supervisor review rather than auto-accepting them. It wasn't a perfect check — a sufficiently motivated person could spoof it — but the honest goal was never to build something unbeatable by a determined bad actor. The honest goal was to raise the cost of fabrication enough that it stopped being the path of least resistance, and to make the pattern visible in aggregate even for the agents who did get past an individual check, because systemic fabrication leaves statistical fingerprints even when any single instance might not.

Rolling all three of those checks together meant every submission, the instant it tried to enter the system, got evaluated against: was the location plausible for the assigned zone, was this a near-duplicate of something recent from the same agent, and did the photo evidence actually carry believable metadata. A submission that failed one of these didn't necessarily get destroyed — fabricating data wasn't the assumed default, and most failures really were honest mistakes or hardware quirks — but it got routed differently, held for a supervisor's judgment rather than flowing straight into the numbers that fed the dashboards and the reports that went up the chain. The crucial shift wasn't the specific rules; it was moving the decision point from "after it's already shaped a report" to "before it's allowed to count as a fact the system treats as settled."

The Other Half: Making Anomalies Find People Instead Of The Other Way Around

Even with good validation at the point of entry, you still have a second, equally hard problem once you're operating at the scale of thousands of agents reporting across hundreds of zones: how does anyone notice when something has gone quietly wrong at the aggregate level, as opposed to any single submission being individually bad? A zone that's supposed to have forty agents submitting roughly twenty records a day each might see its total daily submission count fall by sixty percent over the course of a week, with every individual submission in that week passing all the validation checks cleanly. Nothing about any one record is wrong. But something is clearly wrong with that zone — a supervisor who stopped following up, a batch of devices that broke, a connectivity tower that went down, agents who quietly stopped working and nobody noticed because nobody was required to notice.

The honest first version of how this was being handled, before I got involved in this part of the system, was a weekly manual review where a state-level analyst would open a spreadsheet and eyeball totals by district, looking for numbers that seemed off. This has the obvious problem that human attention doesn't scale with the number of zones, and the less obvious problem that a human skimming a spreadsheet of a few hundred rows a week develops exactly the kind of fatigue where real anomalies get missed because the eye glazes over after row eighty.

What I built instead was a nightly aggregation job that rolled every day's raw submissions up into zone-level, district-level, and state-level summaries, and then — this was the actual point of the exercise — compared each zone's current activity against that same zone's own historical baseline, computed from a trailing window of recent weeks, rather than against some fixed, program-wide target that didn't account for the fact that different zones had wildly different normal operating volumes depending on population density and staffing levels. A zone that normally did fifteen submissions a day falling to seven was just as significant a signal as a zone that normally did two hundred falling to ninety, and a system that compared everyone against one flat program-wide number would miss the first case and overreact to perfectly normal variance in the second.

Getting the sensitivity of that comparison right took real trial and error. My first version flagged a zone whenever its submission count for a day fell more than a fixed percentage below its rolling average, and in the first week of running it live it produced far too many flags — a huge number of zones legitimately have a quiet Monday or a quiet day around a local festival, and flooding coordinators with "anomaly" alerts for ordinary day-to-day noise just taught them to ignore the alerts entirely, which defeats the entire purpose of building the thing. I ended up requiring the dip to persist across a rolling multi-day window rather than tripping on any single day's number, and raising the threshold so that only a meaningfully large and sustained drop — well below half of the historical baseline, sustained for more than one day — actually generated an alert that landed in front of a human. That tuning process, going back and forth with the program coordinators about what felt like a real signal versus noise they already knew to expect, probably mattered more to the eventual usefulness of the system than the mechanics of the comparison itself.

When it worked, it genuinely worked well. One morning that query flagged a cluster of three adjacent zones that had all dropped sharply at the same time, which on its own looked like it might just be a shared local holiday — except it had been flagged for three straight days, past the point a one-day festival explanation would cover. When someone followed up by phone, it turned out a local network tower serving that whole area had gone down and stayed down for most of the week, and nobody above the immediate supervisor level had been told, because the supervisor had assumed it would get fixed quickly and didn't think it was worth escalating. That's the kind of operational problem that a dashboard full of pretty charts would never surface on its own, because the aggregate national numbers barely moved — three zones out of hundreds is a rounding error at the national level. It took a system that was specifically looking for localized, sustained deviation from each zone's own normal pattern to catch something a person skimming a national chart would never have seen.

Living With The Pressure Of Being Wrong In Public

I don't want to make this sound like a purely technical story, because a lot of what made it hard wasn't the logic itself — it was the fact that mistakes here didn't stay contained to a bug tracker. If the validation was too aggressive and rejected honest agents' work, real people doing genuinely difficult jobs under real hardship got penalized for things outside their control, and that bothered me more than almost any production incident I've dealt with before or since, because the stakes were someone's actual livelihood and standing with their supervisor, not just an SLA. If the validation was too lax, bad numbers flowed up into decisions that affected how resources got allocated across a program serving a lot of people who had no idea any of this software existed, let alone that a radius check I'd tuned on a Tuesday afternoon had any bearing on whether their district got attention.

That tension sat with me the whole time I worked on this, and I don't think there's a clean resolution to it beyond being honest that you're always trading off false positives against false negatives and neither direction is free. What I did try to hold onto, as a kind of internal compass when a specific rule's threshold was ambiguous, was a bias toward routing uncertain cases to a human supervisor rather than making an automated, final, silent decision either way. The system's job wasn't to replace judgment; it was to make sure judgment got applied to the right small set of cases instead of being either impossible at scale or applied inconsistently and late.

What I Took Away From It

Looking back, the specific rules — the radius checks, the duplicate window, the EXIF requirement, the baseline comparison — aren't really the point, even though I spent an enormous amount of time getting each one right and wrong before landing somewhere defensible. The point, the thing that's genuinely reshaped how I think about every system I've built since, is that at real scale, with data entering from thousands of sources you don't control and can't fully trust, the hardest engineering problem isn't the dashboard, the API, or the aggregation logic. It's deciding, deliberately and in advance, what counts as trustworthy enough to become a fact your system will treat as settled — and then building the discipline to enforce that decision automatically, at the moment of entry, rather than hoping someone catches the problem downstream after it's already shaped a decision that's hard to undo.

It's a less glamorous lesson than I expected to come out of two years of working on something that touched tens of millions of people, in the sense that there's no clever architecture diagram that captures it. But it's the one I actually use, in every system I design now, whenever I catch myself about to trust a number just because it arrived.

A Problem I Didn't See Coming: Agents Gaming A System They Didn't Design

One thing nobody warned me about going in, and which I only understood after it had already started happening, was that the agents submitting data weren't passive sources feeding a pipeline — they were people with incentives, working under performance targets set by their own supervisors, and incentives shape behavior around any system you build, including ones designed to catch exactly that. A supervisor somewhere up the chain had decided that agents needed to hit a minimum number of verified household visits per week to be considered meeting expectations, and the moment that target existed, a small number of agents started looking for the path of least resistance to hit it rather than the path of doing the actual work.

The clearest example of this showed up almost by accident, through a pattern I noticed while debugging something unrelated. I was looking at submission timestamps for a handful of agents clustered in one district, trying to understand a different anomaly, and I noticed that one particular agent had a suspiciously even spacing between consecutive submissions — almost exactly six minutes apart, for hours at a stretch, day after day. Real fieldwork doesn't look like that. Real fieldwork has long gaps for travel, clusters where several nearby households get visited close together, and genuine variance depending on how long a conversation runs. A perfectly even six-minute cadence looked, to put it bluntly, like someone sitting in one place and submitting fabricated records on a timer rather than actually walking door to door.

I brought this to the program side carefully, because accusing a field worker of fabricating records is a serious thing to get wrong, and I didn't want a statistical pattern alone to end someone's livelihood if there was an innocent explanation. It turned out, after a local follow-up, that the explanation wasn't innocent — the agent had been submitting records for visits that never happened, using details copied from an old paper register from a previous, now-completed round of the program, resubmitted as if they were new. The GPS check hadn't caught it because the agent was physically present in the right village, just not visiting each specific household; the duplicate check hadn't caught it because each fabricated record referenced a different beneficiary; the photo check hadn't caught it because the agent had, apparently, taken a batch of real photos during one actual walk through the village weeks earlier and was recycling them across multiple fabricated submissions, each time with a slightly different timestamp attached.

That incident taught me that cadence and rhythm are a signal in their own right, independent of whether any individual submission passes its own checks, and I ended up adding a lightweight statistical pass that flagged agents whose submission timing was unusually regular compared to the natural variance you'd expect from genuine fieldwork — not as an automatic rejection, because a false accusation here has real human cost, but as a prioritization signal that moved certain agents higher up the list for the periodic manual spot-checks that already existed in the program. It was a humbling reminder that any fixed set of rules, no matter how carefully designed, is a target that adapts to pressure over time, and that you have to keep watching the data with fresh eyes for new patterns rather than assuming the checks you built last year are still catching everything this year.

Explaining The System To People Who Would Never See A Line Of Code

A part of this job I hadn't expected, and which turned out to take up a surprising share of my actual time, was explaining what the validation system was and wasn't doing to people who had no technical background at all — state program directors, field supervisors, sometimes a visiting auditor from an oversight body who wanted to understand how we could claim the numbers were trustworthy. I learned, slowly and through a few uncomfortable meetings where I over-explained the wrong things, that what these audiences needed wasn't a description of the location-matching logic or the duplicate window. What they needed was a plain answer to one question: if a number in this report is wrong, how would we know, and how fast.

I started keeping a short, plain-language one-page explanation that I'd update every time the validation logic changed meaningfully, written entirely without jargon, describing in everyday terms what got checked, what happened when something looked wrong, and who ended up making the final call on an ambiguous case. It wasn't technical documentation in any real sense, and I never would have thought, early in my career, that writing something so simple would be one of the more valuable things I produced on the project. But it became the thing program directors actually referenced in meetings with their own superiors, because it gave them, in their own words, a way to defend the credibility of the numbers without having to understand a single thing about databases, timestamps, or EXIF metadata. Building the trustworthy pipeline turned out to be only half the job. Being able to explain, honestly and simply, why it deserved to be trusted was the other half, and it's a skill I don't think I'd have developed without being forced into those rooms.