Quantitative Trading•寒蝉•6dOpinion

Why I Don't Recommend Individual Quantitative Traders Building Their Own Data Sources

The essence of quantitative investing is data plus factors plus strategy: data is the raw material, factors are the processing, and strategy is the decision-making. The three build on one another, with data at the bottom—factors are calculated from data, and strategies make decisions based on factors. But data quality does not determine the upper limit of a strategy; it determines its lower limit: good data does not mean the strategy will work; bad data means the factors are wrong, and a strategy built on incorrect factors will look completely different in live trading, no matter how good the backtest looks. Data is important—important enough that you should obtain it in the most effortless and reliable way possible, rather than building a pipeline yourself. Let me clarify what “doing it yourself” means here: I mean retaining control of the strategy—stock selection, decision-making, and execution should not be outsourced to a fund. Data acquisition is public infrastructure and does not constitute a competitive advantage, so outsourcing it does no harm. Let me also clarify the scope: what I’m discussing below is the most common scenario for individual quantitative traders—daily-frequency strategies using standard market data and financial statements. The logic of high-frequency trading and alternative data is different, and I’ll return to that boundary at the end. The trap most likely to be overlooked by individual quantitative traders is turning strategy research into an engineering project I’ve seen too many people (including me back then) go down this path: after deciding to do quantitative investing, the first thing they do is build a data pipeline rather than find factors. They crawl market data and financial statements, set up databases, write incremental updates, and then add scheduled jobs and monitoring. The engineering looks impressive, and the sense of accomplishment is satisfying. But this is the trap most likely to be overlooked by individual quantitative traders: turning strategy research into a heavy engineering workload. Spend your time on the loop: the data pipeline is a time black hole What individual quantitative traders should really be doing is running a loop—finding factors, adjusting the strategy, and executing strictly. The faster this loop runs, the faster you improve. Building databases and maintaining data sources are not part of the loop; they are the loop’s fuel supplier. Spending large amounts of time and energy maintaining data sources is not quantitative investing—it’s fixing pipelines. No matter how beautifully you build the pipeline, it won’t make the strategy earn another cent. The deceptive nature of this trap is that it gives you the illusion that “I’m doing quantitative investing”: there’s something to do every day, there’s output every day, the database keeps growing, and the code keeps multiplying. But when you stop and look, your factor ideas are still the same few, the strategy still hasn’t been iterated, and the loop never really got started. Putting the cart before the horse is the most accurate definition of this trap. So I don’t recommend that individual quantitative traders build their own data sources. First, it isn’t because this is technically difficult; it’s because it draws your energy away from the core loop. Second, only then do the pitfalls of building your own data sources come into play. Second: three pitfalls of building your own data source Let me define this first: here, “building your own” means building the entire stack yourself—crawler, database, and cleaning process. Pulling data from an existing source into a database and adding your own adjustment and cleaning logic, which is the “semi-self-built” approach many people realistically choose, has manageable costs and does not count as a pitfall. Truly building it yourself has three pitfalls, each of which is enough to make an individual quantitative trader fail: Pitfall one: if the data is wrong, you won’t know. Adjustments are handled incorrectly, survivorship bias hasn’t been removed, financial statements include data from before their publication dates, or limit-up signals are calculated when the stock could not actually be bought. These errors won’t throw an exception or issue a warning. They simply make your backtest curve look good. You think the strategy works, when it’s actually the data deceiving you. Then you start tuning parameters, going further and further off track, until you’re fitting parameters to data errors. Pitfall two: maintenance never ends. Anti-scraping measures get upgraded, interfaces change, financial statement fields get adjusted, and ex-dividend and ex-rights rules have to be supplemented manually. The true cost of building your own system was never “the two weeks spent writing the crawler”; it was the ongoing consumption of every day afterward. Institutions have people dedicated to this. Individual traders only have the few hours in the evening—the data pipeline will quietly eat them all. Pitfall three: a one-person system has no backup. Institutional data pipelines have reviews, tests, and alerts; when something breaks, someone notices and fixes it. An individual writes it, tests it, and deploys it alone. If a rule is wrong, no one stops you; if something fails silently, no one alerts you. A single point of failure isn’t a matter of probability; it’s a matter of time. One more clarification: using an existing source doesn’t mean you don’t need to understand data. You still need to know where pitfalls such as adjustments and survivorship bias lie so that you can judge whether the data is correct. An existing source saves you the process of discovering those pitfalls through experience; it doesn’t replace your understanding. How to choose a data source: the standard is whether it pays for itself So should you buy a data source, and which one? Don’t rush to compare feature lists. My standard is simple: can your strategy earn the money back? At the beginning, free sources such as akshare and baostock are perfectly adequate, but you need to understand what you’re getting: they are essentially community-maintained crawlers, with interfaces that change frequently and field quality that varies. Using them isn’t risk-free; it simply outsources maintenance to the community, and the community doesn’t guarantee availability. Tushare Pro costs 500 yuan a year, which is enough for an individual working on daily strategies. What does 500 yuan mean? If the strategy really works, even if it only adds 0.1% annualized returns, 100,000 yuan of capital would generate 100 yuan in a year—the data fee is such a small line item in your ledger that it shouldn’t become a decision variable. If you don’t need real-time data, there’s an even cheaper option: spend a few yuan on a shared account, download the historical data once, and store it locally for gradual research. This is still “using an existing source”—you’re simply replacing an online subscription with a local snapshot. Keep your own cleaning and adjustment logic, and conduct your research the same way as before. This doesn’t violate the principle above: you haven’t spent time maintaining a pipeline; you’ve simply obtained the data. Whether to spend money or save it comes down to features and data accuracy. There’s just one question: can your strategy earn the money back? If it can, buy it. If it can’t, the core issue isn’t the data fee either—saving that small amount won’t rescue a strategy without alpha. When building your own data source makes sense Building your own data source isn’t absolutely forbidden; it only makes sense within certain boundaries: Data that commercial sources don’t cover, such as tick-by-tick block trade details or certain types of alternative data. If you can’t buy it, build it yourself. Individually processed data, such as factors and derived indicators you design yourself. This isn’t building your own data source; it’s building your own feature engineering. You should do it, and you must do it yourself. Purely for learning technology. Go ahead, but understand that you’re paying to learn, not to save money—and certainly not for the sake of your strategy. There’s just one criterion: if you’re building your own system to make “the data more reliable,” you’ll probably be disappointed. If you’re doing it to “obtain something commercial sources don’t have,” then it genuinely makes sense. Give your time back to the loop The deciding factor for individual quantitative trading is the speed of the “find factors → tune the strategy → execute strictly” loop. Data is water, and strategy is the fish. Your job is to use clean, readily available water to raise fish that can swim as quickly as possible, rather than spending two years digging your own well first. Data is important, which is exactly why you shouldn’t maintain it yourself. Outsource maintenance to mature data sources and keep your time for yourself—for the loop that truly determines your returns. As for high-frequency and alternative data, that’s a different story: “you can’t buy it” is the norm there, and building your own system genuinely becomes necessary. But on the battlefield of daily strategies and standard data, outsourcing the pipeline and preserving time for the loop offers the highest return on investment.