Home/Services/Data, Analytics & AI
Service 06 — Data, Analytics & AIData platforms that stay cheap as they grow.
Most analytics platforms are affordable at launch and painful at year two. The difference is almost always partitioning, file layout and governance decided early — not the query engine you chose.
Get the boring decisions right first
Storage layout, partitioning strategy, file format and table format are the decisions that determine what your platform costs in three years. They are also the ones most likely to be made in week one by whoever was available.
- S3 zone layout — raw, curated, consumption — with lifecycle policy per zone
- Open table formats (Iceberg) where schema evolution and time travel earn their keep
- Partition strategy derived from actual query patterns, not from a convention
- Compaction and file-sizing jobs, because small files are the single biggest cost trap
- Glue Data Catalog as the single catalogue, with Lake Formation permissions
- Query layer chosen per workload: Athena, Redshift Serverless, or both
Ingestion that fails loudly
The worst failure mode in data is not an outage. It is a pipeline that keeps running and quietly produces wrong numbers that somebody uses in a board paper.
Sources
Operational databases via DMS change data capture, SaaS APIs, event streams on Kinesis or MSK, and file drops with schema validation at the door.
Orchestration
Step Functions, Managed Airflow or dbt — matched to your team's actual skills rather than to whatever is fashionable this year.
Data quality
Freshness, volume, schema and business-rule checks that halt the pipeline and page someone, instead of writing a warning to a log.
Idempotency
Reprocessing a day should produce the same result. Sounds obvious; is untrue in most pipelines we inherit.
Backfills
A documented, tested, bounded backfill procedure — so a correction does not become a three-day incident with a surprise invoice.
Lineage
Column-level where it matters, so "where did this number come from" has an answer that does not require the person who left last year.
Access control that survives an audit
Australian privacy obligations do not stop at the application boundary. A data lake full of production personal information with broad read access is a Notifiable Data Breach waiting for a misconfigured role.
- PII classification and tagging at ingestion, not retrospectively
- Lake Formation row- and column-level permissions tied to Identity Center groups
- Masking and tokenisation for non-production and analyst access
- Retention policy enforced by lifecycle rules, with legal-hold exceptions
- Access logs retained separately from the accounts being audited
- Data residency held in Australian regions unless you direct otherwise
A question worth asking your team
If an analyst account were compromised tomorrow, exactly which records would be readable, and would you be able to demonstrate that answer to a regulator inside 72 hours?
Most organisations cannot answer either half. Getting to where you can is usually two to four weeks of work, and it is the highest-return security investment a data platform can make.
Make analytics spend scale with value, not table size
Partition
On the columns actually filtered. A date partition nobody filters on costs you money and buys nothing.
Compact
Target 128–512 MB files. Small-file overhead dominates cost on nearly every lake we inherit.
Columnarise
Parquet with the right compression codec. Usually the single largest one-off reduction available.
Attribute
Query cost per team and per dashboard. The top three consumers are almost never who anyone expects.
The dashboard nobody opens
On one engagement, a single hourly-refresh dashboard accounted for 22% of total Athena spend. It had not been opened by a human in four months. Attribution is what makes findings like that possible.
Enablement with guardrails, before the pilot becomes production
Generative AI projects tend to reach production faster than the controls around them. The two questions that matter — where does the data go, and what does it cost at scale — are usually answered late.
- Amazon Bedrock foundations: model access policy, private connectivity, region choice
- Retrieval architecture with your own data, including access-control propagation
- Prompt and output logging with retention aligned to your privacy obligations
- Guardrails for content, PII redaction and refusal behaviour
- Token spend attribution and budget alerts per application and per team
- Evaluation harness so model or prompt changes are measured, not vibes-tested
SageMaker & classical ML
Feature storage, training pipelines, model registry, endpoint hosting with autoscaling, and the monitoring that catches drift before a business user does. Frequently the right answer where a foundation model is being used for something a small trained model does more cheaply and more predictably.
What we will tell you
Where a GenAI feature will not survive contact with your data quality, we say so during design rather than after the pilot. The most common blocker is not the model — it is that the underlying data was never governed well enough to retrieve from.
Data questions
Redshift or Athena?
Athena for exploratory and intermittent workloads where you pay per query. Redshift where you have sustained concurrency, complex joins and predictable workloads that benefit from a warm cluster. Plenty of platforms run both over the same catalogue, and that is a legitimate answer rather than indecision.
Can you work with Snowflake or Databricks?
Yes. We manage the AWS foundations underneath — networking, identity, storage, cost and security — and integrate cleanly. We will not pretend a migration off them is free when your team has years of investment there.
Our data platform costs more than it returns. Where do we start?
Attribution. Until you know which queries, dashboards and teams drive the spend, every optimisation is guesswork. That measurement typically takes under a week and frequently pays for the whole engagement on its own.
Do you build dashboards and reports?
We build the platform, the pipelines and the governance. We can stand up QuickSight and a reference set of dashboards, but ongoing analytics and reporting content is better owned by people who understand your business. We will say so rather than bill for it.
Find out what your analytics actually cost
A week of attribution work usually reframes the entire conversation — and often pays for itself before the engagement starts.