From Field Devices to AWS: DevOps for a Harvest Data Platform
AppRecode helped modernize field data collection, local synchronization, and cloud processing for an agricultural operation with approximately 200 people in field operations—replacing fragmented legacy synchronization with an integrated AWS platform.
About the Project
Sector & Scale
Agricultural production, sorting, and wholesale logistics involving approximately 200 field and warehouse operators across remote regions.
Challenges
Intermittent connectivity in orchards, delayed batch transfers via legacy SQL Server (~120 GB), and manual reconciliation slowing buyer-facing publications.
Our Approach
0.25 FTE lead architect + full-time mid-level engineer during active months, scaling to a dedicated part-time support engineer for stabilization.
Disconnect Between Field Harvest & Commercial Velocity
Agricultural harvesting requires continuous precision: field teams scan bins, evaluate crop weights, and print track-and-trace tags amidst harsh dust, moisture, and fluctuating cell coverage. Historically, field captures relied on detached local storage units with unpredictable manual sync passes at day’s end.
These delayed uploads forced wholesale dispatchers and commercial buyers to wait for overnight batch updates before negotiating dynamic sales or dispatching freight trucks. A single network disruption or corrupted SQL Server table could block inventory visibility across entire distribution channels.
Key Results
4× faster queries and platform response
120 → 4 GB relational database size
Part-time support after active delivery
Tech Stack
- AWS
- Raspberry Pi devices, AWS IoT Greengrass, local PostgreSQL
- ECS on EC2
- AWS Lambda, SNS, SQS, EventBridge, Step Functions
- RDS for PostgreSQL, Amazon S3
- Terraform, Terragrunt
- Serverless Framework
- GitHub Actions
- Amazon CloudWatch, CloudWatch Logs, Grafana
Numbers
Deployment time: –70%
Uptime: 99.995%
Incident Reduction: –146.5%
Release Frequency: the same, but the time reduced from hours to minutes
MTTR <1 day
Want deployments to be 2-3× faster?
We’ll automate build/test/deploy and remove the manual steps slowing your team down.
Three-Zone Logical Architecture
AppRecode structured the operational environment into three decoupled tiers: the physical harvest perimeter (“Field”), on-premise packing and weighline facilities (“Local operations”), and a centralized high-availability infrastructure (“AWS”).
Edge nodes communicate securely using AWS IoT Greengrass deployments, buffering records on local NVMe-backed micro-instances when cellular towers experience signal fade. When backhaul connectivity clears, events fan out across asynchronous cloud workers.
Zone A: Field
Raspberry Pi field devices. Barcode scanning, weigh scales, pallet labels.
Zone B: Local Operations
- Local operational line. Grading conveyor interfaces, line supervisors.
- Local PostgreSQL / buffering. FIFO storage queues & device staging buffers.
Zone C: AWS Cloud
- Asynchronous processing. Lambda, SNS, SQS, EventBridge, Step Functions
- Compute Infrastructure. ECS on EC2
- RDS for Postgre. SQL Core tabular domain model (~4 GB)
- S3: attachments and files. Binary inspection photos, PDF harvest manifests
Edge Decoupling & Network Fault Tolerance
Cellular modems operating along valley rows frequently experience dead zones. The edge software architecture guarantees that data collectors never block the physical sorting line during internet dropouts.
Database Modernization & Storage Separation
The original database had grown organically over years into an unwieldy monolith. Roughly 150 tables were defined in Microsoft SQL Server, yet telemetry revealed only ~50-60 tables were actively referenced in daily operations. More critically, heavy binary attachments (high-resolution yield photos, scanned bills of lading, packaging proofs) were stored directly in database BLOB columns, inflating table sizes and causing extreme memory pressure.
Workflow Transformation
Active infrastructure development
Core IaC creation, edge runtime design, migration of the SQL Server monolith to PostgreSQL, and setup of all four CI/CD delivery lanes.
- Lead Architect: 0.25 FTE
- DevOps Engineer: Full-time mid-level
- Milestone: Staging validation before the agreed release.
Support and additional features
Post-cutover operational observation, addressing minor edge-case buffering anomalies, knowledge transfer to client maintainers, and adding non-critical metrics.
- Lead Architect: Advisory on-demand
- Engineering Staff: Part-time engineer
- Workload: Lower ongoing workload with self-sufficient client team.
Is your infrastructure growing faster than your processes?
We’ll build a DevOps foundation that scales with your product - not against it.
FAQ
Field data reached the business too late. Harvest records were synced by hand at the end of the day, so dispatchers and buyers waited for overnight updates before they could sell produce or send trucks. The new platform sends data to the cloud automatically and keeps it safe when the connection drops, so the commercial team works from current figures.
They keep working when the signal drops. Scans, weights and labels are saved on the device and in local storage, then uploaded automatically when the connection returns. Nobody has to wait for coverage or run a manual sync at the end of the day.
The serverless part that receives field data costs about $3–4 a month (Lambda requests and execution, excluding S3 storage). Shrinking the database also lowered CPU load enough to run on a smaller server. The application runs on dedicated EC2 instances, chosen partly for predictable cost.
Engineering and business teams have separate views. Technical staff monitor system health, queues and capacity in Amazon CloudWatch. Operations leads use Grafana dashboards showing harvested volume, daily revenue and week-over-week trends.
Updates to the Raspberry Pi devices are published through AWS IoT Greengrass, which reports the deployment status of each device. If a release causes problems, it can be rolled back.
Rate Apprecode as Your Partner
8 ratings, average 4.9 out of 5
Comments
Loading...
