Amazing Devs

AI Project Scoping: A Practical Guide for Tech Leaders

AI Project Scoping: A Practical Guide for Tech Leaders

AI project scoping title card illustration

Scoping an AI project means deciding, before anyone writes production code, what problem you are solving, whether AI is the right tool, what data and people it needs, how you will measure success, and what is deliberately out of bounds. Done well, it takes days or a few weeks and saves months. Done badly, you get a demo that impresses a steering committee and never ships.

This guide walks through the scoping process in the order we recommend working through it, with a scope template, a worked example and the questions that tend to kill projects early.

Amazing Devs
Add Engineering Capacity to Your AI Pilot
Amazing Devs connects businesses with skilled nearshore developers from Brazil, with recruitment focused on technical ability, cultural fit and business alignment.

Explore nearshore developers

TL;DR

  • Start with a business problem and a measurable outcome, not a model or a vendor. Then test whether a simpler non-AI approach already solves it.
  • Check data, feasibility, cost drivers and risk before committing budget. Most surprises in AI projects come from data, not algorithms.
  • Write a scope statement with deliverables, exclusions, constraints and acceptance criteria, and agree thresholds for quality, cost and latency in advance.
  • Prove the idea with a narrow pilot on representative data, then continue, pivot or stop based on the numbers you set up front.
  • Name the roles and the engineers you need while you scope, because capacity is the most common hidden constraint.

Table of Contents

Why scoping decides whether an AI project survives

RAND researchers interviewed data scientists and engineers about why AI projects fail. One of their findings is that organizations sometimes chase the newest technology instead of solving a real problem for the people who will use it, and another is that many lack the infrastructure to manage data and deploy finished models (RAND). Both are scoping failures. They are visible on day one if someone asks the right questions.

AI projects differ from ordinary software projects in three ways that make scoping harder:

  1. The outcome is uncertain. You can specify a login screen. You cannot specify that a model will reach a given accuracy on your data until you test it.
  2. Data is a dependency, not an input. Its quality, ownership and legal status shape what is possible.
  3. Costs are variable. Inference, prompt length, retrieval and monitoring all scale with usage, so a cheap demo can become an expensive product.

Good scoping does not remove that uncertainty. It makes the uncertainty small, named and cheap to resolve.

Step 1: Define the problem in business terms

Microsoft’s Engineering Fundamentals Playbook says the envisioning phase should pin down the problem domain, the business objective behind it, the intended use, performance measures, available data, team capabilities, and a shared scope with next steps (Microsoft). It also asks teams to state why the problem needs solving, who will use the solution and what function it must perform, then document it so every stakeholder shares the same understanding.

A useful problem statement is specific enough to fail. Compare:

  • Weak: “Use AI to improve customer support.”
  • Strong: “Reduce the average first-response time on tier-one billing tickets from several hours to under 30 minutes, without raising the escalation rate.”

The second version names a user, a process, a number and a guardrail. You can tell whether you have achieved it.

The four Ws

If your team likes a quick framework, the four Ws are a decent starting prompt. DataCamp describes them as Who (the stakeholders and beneficiaries), What (the problem, defined and validated), Where (the situations and contexts in which it occurs) and Why (the benefits and value of solving it) (DataCamp). Treat them as a warm-up, not a scope. A real scope still needs metrics, data and constraints.

Four Ws framework for defining an AI problem

Ask the people who live with the problem

Andrew Ng’s scoping approach, as summarized in a 2021 write-up of his talk, starts from a business problem rather than an AI problem and suggests asking domain experts which important things work poorly and why (summary of his talk). Operations leads, support managers and analysts will often name three problems you had not considered, and one of them is usually a better first project than the one on your roadmap.

Step 2: Decide whether AI is the right tool

This is the step teams skip, and it is the cheapest one to take seriously.

Google for Developers advises stating the product goal first, then deciding whether predictive ML, generative AI or a non-ML solution fits best. It also recommends building or optimizing a non-ML baseline before adopting ML, then comparing the expected quality gain against cost, maintenance and your organization’s ability to run it (Google for Developers). PMI’s CPMAI guidance makes a similar point: evaluate AI fit against simpler alternatives and set KPIs and ROI expectations before development starts (PMI).

A practical filter:

Question If the answer is yes If the answer is no
Can rules, search or a spreadsheet hit the target? Ship that and revisit AI later Continue scoping
Is there an example set of correct answers, or can experts produce one? Evaluation is possible Fix this before anything else
Does the task tolerate occasional errors, with a human check where it matters? Good candidate Narrow the use or add review steps
Will the gain justify ongoing run and maintenance costs? Proceed Park it

Andrew Ng’s reminder is blunt: after you understand the problem, brainstorm several approaches, because sometimes no good AI solution exists (summary of his talk). Finding that out in week one is a win.

Step 3: Set success metrics you can calculate

AWS Prescriptive Guidance says success metrics should be defined with stakeholders, tied to business outcomes, quantifiable, and revisited as planning reveals better or cheaper measures. It also recommends specifying how each metric is calculated and setting an initial target within a defined timeframe (AWS Prescriptive Guidance).

Pair two kinds of metrics, a practice Andrew Ng also recommends: a model metric such as accuracy, and a business metric such as engagement or revenue. If you cannot set reasonable milestones, he says, you probably need more understanding of the problem (summary of his talk).

For the support example above, a metric set might look like this:

  • Model metric: share of drafted replies that agents accept with no or minor edits, measured on a labeled sample of past tickets.
  • Business metric: median first-response time on tier-one billing tickets.
  • Guardrail: escalation rate and customer satisfaction must not worsen.
  • Cost metric: all-in cost per resolved ticket, including inference.
  • Threshold: the numbers at which you continue, pivot or stop, written down before the pilot.

That last line matters. Teams that set the thresholds after seeing results tend to talk themselves into the answer they wanted.

Step 4: Assess data and technical feasibility

Google lists the dimensions of ML feasibility as data availability, problem difficulty, prediction quality, technical requirements and cost. Its checklist asks whether labeled examples are sufficient and affordable, whether the features you rely on will exist at serving time, and whether regulations govern how you acquire and use the data (Google for Developers).

PMI’s data-understanding phase adds practical detail: identify the data you need, check availability and access, record owners and formats, and assess accuracy, completeness and consistency, along with the effort to prepare it and build the pipelines (PMI).

Turn that into a one-page data audit:

  1. Where does the data live, and who owns it? Name a person, not a system.
  2. Can we legally and contractually use it for this purpose? Involve legal early if it contains personal or customer data.
  3. How clean is it? Pull a sample of a few hundred records and read them. Nothing replaces this.
  4. Will it exist at runtime? A feature that is only populated after the outcome is known will make your offline results look great and your product useless.
  5. Who can label or validate it? Domain experts are scarce and their time is part of the budget.

For technical feasibility, Ng suggests three routes: published work, how competitors approach the problem, or a quick proof of concept. He suggests assessing value with domain experts (summary of his talk).

Step 5: Count the costs and the risks

For generative AI, the AWS Well-Architected Generative AI Lens says scoping should weigh technical and organizational feasibility, business and technology risks, and the availability and quality of data needed to customize the model. It names cost drivers worth modeling early: prompt lengths, data architecture and access patterns, model selection and orchestration (AWS).

Two practical habits help here.

Model cost at three volumes. Estimate run cost at pilot volume, expected production volume and ten times that. If the unit economics break at the second number, you have found a scoping problem rather than an engineering one. When your design involves training or fine-tuning, it helps to understand the cost drivers of LLM training so you can tell whether a given budget buys a fine-tune, a retrieval layer or just API calls.

Keep a risk register from day one. List what could go wrong (bad data, model errors in front of customers, security exposure, regulatory trouble, a key engineer leaving), give each an owner and a trigger, and review it every sprint. A step-by-step guide to building an AI risk register is a good template for teams doing this for the first time. The PMI-NASSCOM playbook backs the underlying idea: manage risk throughout the lifecycle with defined triggers, ownership, escalation paths and timely communication about time and cost risks (PMI-NASSCOM).

Step 5: Count the costs and the risks — overview diagram

Step 6: Write the scope statement

Asana’s guidance on scope statements says a strong one specifies goals and objectives, tangible deliverables, what is included and excluded, constraints such as budget, timeline and resources, and acceptance criteria, meaning the standards deliverables must meet for stakeholders to approve them. It adds that requirements should be validated with stakeholders before the scope is final (Asana).

Here is how that maps onto an AI project:

Scope element What to write for an AI project
Objective One sentence with the user, the outcome and the number
Deliverables A working pilot, an evaluation report, an architecture note, a go/no-go recommendation
In scope The specific use case, data sources and user group
Out of scope Other languages, other departments, full automation, model training from scratch
Constraints Budget ceiling, timeline, data access limits, security and compliance rules
Acceptance criteria Quality, cost and latency thresholds, tested on a held-out evaluation set
Dependencies Data owners, labelers, security review, integration teams
Assumptions What you believe is true and how you will check it

The exclusions row is the one that prevents scope creep. In AI work, creep often arrives as “while we’re at it, can it also…” after a promising demo. Writing down what you will not do gives you something to point at.

Budget people, not just compute

Andrew Ng’s resource step includes data, personnel, time, integrations and support from other teams, and he describes scoping as iterative: when new evidence makes an approach infeasible, you go back and revise (summary of his talk). In our experience, people are the constraint that sinks timelines. A scope that needs two ML engineers and a data engineer for eight weeks is only a scope if you can actually staff it. If you cannot, that belongs in the document as a dependency.

Step 7: Validate with a narrow pilot

AWS guidance for generative AI proofs of concept says to choose a narrow, high-value use case and test the simplest credible architecture with representative data and domain-expert evaluation. Validate technical feasibility, business usefulness, integration, safety, latency and unit economics in short iterations, define quality, cost and latency thresholds in advance, and continue, pivot or stop based on the results (AWS Prescriptive Guidance). The same page asks for validation of privacy and security, data inventory and access, data lineage, and evaluation datasets with expert-validated reference answers.

A pilot plan that follows this guidance fits on one page:

  1. Pick one use case and one user group. Not three.
  2. Build the evaluation set first. Have experts write or approve correct answers before you test any model.
  3. Start with the simplest architecture that could plausibly meet the thresholds, such as a hosted model with retrieval, before anything custom.
  4. Run short cycles of one to two weeks, each ending in a measured result.
  5. Hold a decision meeting against the pre-agreed thresholds: continue, pivot or stop.

Quality assurance belongs in every cycle, not only at the end. The PMI-NASSCOM playbook recommends recurring checks on data quality, data security and model accuracy at every stage rather than only during final model testing (PMI-NASSCOM). If you need extra hands for test design and regression suites, our nearshore QA contractors cover that.

A worked example: scoping a contract-review assistant

Imagine a mid-sized software company whose legal team spends hours flagging non-standard clauses in vendor contracts. Here is how the scoping might go (the numbers are illustrative, not from a real client).

  • Problem: legal reviewers spend most of their time on routine contracts, so review of strategic deals waits.
  • AI fit test: a clause checklist plus keyword search catches some issues. The team measures how many it catches on 100 past contracts. If the gap is large, AI earns its place.
  • Metrics: recall of flagged clauses against lawyer-labeled contracts, reviewer minutes per contract, and a hard guardrail that no high-risk clause is missed in the test set.
  • Data audit: are the contracts stored consistently, who owns them, and do any carry confidentiality terms that restrict processing by third-party model providers?
  • Scope: English-language vendor agreements only, human sign-off required, no automated redlining.
  • Pilot: four weeks, 100 contracts, thresholds set in advance for recall, cost per contract and response time.

Notice what the example does not include: a model choice. That comes after the problem, data and thresholds are fixed.

Step 8: Staff the project and name the roles

Scoping ends with a team. You need someone who owns the business outcome, someone who owns the data, engineers who can build and evaluate, and a reviewer for risk and compliance. Our guide on fixing AI team roles with RACI and the NIST framework shows how to make those responsibilities explicit, and our playbook for building an AI team in weeks covers how to get people in the seats quickly. If you are hiring, the AI engineer skills to screen first will save you interview time, and our piece on what AI-native development means for engineering teams explains how those engineers actually work.

Our perspective: scope for capacity, not just for ideas

Most scoping advice assumes the team already exists. In the conversations we have with startups and scale-ups, the opposite is more common: the use case is clear, the budget is approved, and the plan stalls because nobody can say who will build it. So we scope with three questions that sit beside the technical ones.

First, what skills does this pilot need in the first four weeks, and which can wait? A pilot rarely needs a full platform team. It often needs one strong AI-enabled engineer who can wire up retrieval, evaluation and a thin interface, plus someone to build the test harness.

Second, how much overlap with your working hours do you need? Evaluation loops and stakeholder reviews go faster when engineers are in your time zone, which is one reason US teams look at Brazil. Our notes on managing time zones cover how to keep that coordination cheap.

Third, can the scope flex if the pilot says stop? A stop decision should release people cleanly, not strand a hired team. Flexible engagements make honest go/no-go decisions easier, because nobody is defending headcount.

Our recruitment process tests technical skill and cultural fit, because an engineer who can discuss tradeoffs with your product owner is worth more in a scoping phase than one who only writes code.

— Gabriel

Need engineers for the pilot?

If your scope is ready and the gap is people, Amazing Devs places pre-vetted Brazilian developers with US companies, handling sourcing, technical and cultural assessment, and contract management. You can see how it works on our nearshore staff augmentation page, or look at the Team Extension Model if you want engineers who work as part of your own team. For a sense of budget, see our guide to Brazil developer rates.

Common scoping mistakes to avoid

  • Starting with the model. “We need an agent” is a solution, not a problem.
  • Skipping the baseline. Without a non-AI comparison you cannot show the AI added value.
  • Evaluating on friendly data. Test on messy, representative inputs, including the ugly ones.
  • Leaving thresholds vague. “Good enough” is not a number.
  • Ignoring run costs. A pilot that costs little to build can cost a lot to operate.
  • Forgetting the humans. Adoption, review steps and training are part of the scope.
  • Treating the scope as final. It is a living document. Revise it when evidence changes, and record why.

FAQ

What are the four Ws in AI problem scoping?

They are Who, What, Where and Why. Who covers the affected stakeholders, What defines and validates the problem, Where identifies the contexts in which it occurs, and Why states the expected value of solving it (DataCamp). They are a good first pass at the problem statement, but you still need metrics, data and constraints.

How long should AI project scoping take?

There is no standard figure, and it depends on how many stakeholders and data sources are involved. For a focused use case, many teams spend a few weeks on problem framing, a data audit and a scope document, then move to a short pilot. If scoping drags on for months, the use case is probably too broad.

How do I know if AI is the right solution?

Build or at least estimate a non-AI baseline first, as Google recommends, then compare the expected quality gain against the cost, maintenance and your team’s ability to run it (Google for Developers). If a simple rule or search gets close to your target, start there.

What should an AI scope document include?

Objective and success metrics, deliverables, what is in and out of scope, constraints on budget, timeline and data, acceptance criteria, dependencies and assumptions. Add the quality, cost and latency thresholds for the continue, pivot or stop decision.

Who should be involved in scoping?

A business owner who cares about the outcome, domain experts who can judge correctness, the person who owns the data, engineers who can assess feasibility, and someone responsible for security and compliance. Bring them in before the scope is final, since requirements should be validated with stakeholders first.

Sources