The Real Challenges of Building My First AI App from Zero

Painless AI Lab – A Hong Kong Practitioner’s Real-World Journey (Part 8)

In my previous post, I shared why I decided to build “Find the Right Stock” and gave an overview of the journey.

Today, I want to pull back the curtain and be entirely transparent about the other side: the actual hurdles I faced while building this system from scratch. If you are thinking about creating your own custom AI tool or automation workflow, I hope my firsthand struggles give you a realistic picture of what it takes.

1. Defining the Pain Point and the Solution

With the current economic downturn, many people like me are looking for reliable ways to grow their wealth. Investing spare money is a sound strategy, but doing it blindly is dangerous. I didn’t want to rely on guesswork, so I set out to build an application that could generate structured financial reports to help me analyze companies objectively. Defining this clear scope was my first essential step.

2. Navigating the Noise of Market Variables

Stock prices shift rapidly, driven by a complex mix of macroeconomics, industry health, and company-specific data. My immediate challenge was figuring out which exact factors were critical for evaluating a business and spotting market risks.

To bridge this knowledge gap, I dug into investment literature, analyzed reputable financial channels, and used AI to brainstorm core frameworks. This research formed the foundational logic for my custom system prompt.

The Lesson: You don’t need to be an expert to start. You build the expertise while you build the tool.

3. Sourcing Reliable Data Streams

• The Problem: For a financial tool to be useful, timing is everything. Connecting to financial APIs sounded straightforward initially, but handling the data pipelines was more technical than I anticipated. Furthermore, scaling up API usage brings added data costs.
• The Solution: I directed my coding assistant to write the precise scripts to fetch and parse this live data seamlessly.

4. Overcoming “Tool Fatigue”

• The Problem: The current landscape is flooded with endless frameworks, platforms, and models. For a beginner, trying to figure out which combination is stable yet simple is incredibly overwhelming.
• The Solution: I made a deliberate choice to keep my tech stack lean. I committed to Ollama for running models locally (which protected my privacy) and paired it with Vercel for seamless hosting. Stripping away the complexity early on saved my sanity.

5. Fighting Hallucinations and Generic Content

• The Problem: Having an idea is easy; making a product reliably execute that idea is the real battle. In the beginning, the system kept spitting out generic, surface-level financial advice. Worse still, it would occasionally “hallucinate” or misinterpret data trends.
• The Solution: This is where I invested the most hours. I rigorously iterated on my “Financial Logic Brief” (my core system prompt). I stress-tested it across dozens of different stock tickers, continuously refining the constraints until the output became strictly factual, analytical, and honest.

6. Hitting Hardware Bottlenecks

• The Problem: Running robust local models entirely on a standard office laptop is heavy work. My system ran incredibly slow and suffered frequent crashes early on.
• The Solution: I pivoted by starting with highly optimized, smaller open-source models. When I needed deeper analytical power, I designed a hybrid approach—using lightweight local models for basic tasks and routing heavier processing to secure cloud models.

7. Making the App Truly Useful (Not Just a Novelty)

• The Problem: It is incredibly easy to build something that looks cool in a terminal but offers very little practical utility. I constantly had to check my ego and ask: “Would I actually open this tool to check a stock before deploying my own hard-earned money?”
• The Solution: I stripped out the fluff and focused the UI entirely on actionable insights. I also handed the tool over to a few trusted friends, asking for brutal feedback on whether the generated financial reports were genuinely easy to read and understand.

8. Sustaining Momentum When Progress Stalled

• The Problem: There were plenty of late nights where the code broke, the data failed to fetch, and I wanted to walk away from the project entirely.
• The Solution: I anchored myself to my original goal: building a reliable filter for my own investments and learning Prompt Engineering through direct, practical application. Celebrating the tiny milestones—like the first time a report formatted perfectly—is what kept me moving forward.

Final Thoughts

Building a custom tool from zero is never as seamless as a polished YouTube tutorial makes it seem. You will hit technical roadblocks, face massive knowledge gaps, and experience real frustration.

But dealing with those friction points is exactly how real upskilling happens. Every error forced me to think more logically. If you are preparing to build your own custom AI workflow, remember these three rules:

1. Start small: Solve one micro-problem first.
2. Focus on utility: Building something useful beats building something complicated.
3. Embrace iteration: Expect it to fail the first ten times, and refine it until it works.
4. If you meet a problem fail to solve, take a deep breath and take a rest first

Question for you:

Have you ever built a personal tool or automated system to manage your work or portfolio?

How do you keep human emotion out of your strategy when the market gets volatile?

Let’s swap notes in the comments below.