I Built a Local AI Financial Advisor to Manage Margin, Taxes, and Investments

Summary

After leaving Google, the author faced more than half a million dollars in margin debt spread across a complex set of brokerage, retirement, health, crypto, credit-card, and private-company accounts. Existing finance apps could aggregate much of the data, but did not provide a sufficiently complete second opinion on leverage, tax consequences, share sales, or a possible move abroad. He therefore built a laptop-based system that combines a local ledger with AI agents for personal-finance summaries, spending analysis, investment research, trading review, tax planning, and broader financial planning. The system uses `ledger.json` for positions, cost basis, cash, debt, provenance, and dates; `facts.json` for verified personal facts; `rules.json` for limits; and a decision memory file to preserve confirmed facts and choices. Account data is gathered through read-only APIs, exports, or computer use after the author handles login and two-factor authentication. The agent does not move money or change account settings, but account contents are still sent to model providers, so local storage does not mean offline processing. The author says the system helped reduce his debt by about 70%, while stressing that ledger errors, missing cost basis, bad exchange rates, and incorrect model interpretations require checks against statements and tax returns. He found stronger reasoning models more reliable for complex work than faster models, although even advanced systems make tax and financial mistakes. Code performs calculations and enforces constraints, while models explain results and the author makes decisions. The article warns against autonomous trading: model recommendations changed frequently, public trading claims are difficult to verify, and real-money model-trading experiments suffered losses. The seven open-source skills on GitHub cover setup, data gathering, research, portfolio review, planning, memory, and dashboard checks. The author expects AI to have more value in financial behavior, tax management, and routine assistance than in stock picking, but identifies privacy, prompt or account compromise, data-access rules, and cross-border tax complexity as continuing risks.