Hello
It's been a while since I last wrote.
I haven't been able to pay much attention to AI news or learning for the past few months because I've been so busy with a project...
The current project is a BI project, and we are working on converting the existing ETL to ELT using DBT.
I'd like to share some of my thoughts on using Claude as the main tool for this conversion work.
Limitations of LLMs
LLMs are not as suitable for analyzing and converting existing systems as one might think.
- The inconsistency in the analysis results is thought to be the cause.
- It seems that writing a mechanical analysis program based on rules would yield better results.
LLMs have little knowledge of commercial software.
- Most BI commercial tools have closed information, so there is no inherent knowledge.
- During the project, it is necessary to teach and accumulate knowledge about LLMs.
- It is essential to document and build a database so that the accumulated knowledge can be effectively utilized by LLMs.
Don't trust the results of LLM analysis.
- Even after going through the above process, the analysis results of LLMs are not perfect.
- Users must always be skeptical of the results of LLMs and track them to the end to ensure that the correct results are obtained.
LLMs make hasty judgments.
- They tend to extrapolate partial results and use uncertain interpretations as conclusions.
- Always keep in mind the user's precautions mentioned above when instructing LLMs on tasks.
LLMs in the Project
Underestimation of the limitations of item 1 by clients and project members.
- Clients and project members who are unaware of the process of deriving analysis data often mistakenly believe that the results are easily obtained or have excessive confidence in them.
- It is necessary to periodically remind people about the limitations of AI.
Caution against excessive function and scope expansion due to LLM use.
- When instructed to analyze a specific issue, LLMs tend to expand their work by checking for similar cases.
- Users also tend to expand the scope of processing based on the LLM's response.
- Although the processing capability of LLMs is excellent and the initial expansion of work may not seem burdensome, subsequent verification and assimilation of these tasks will be the responsibility of the user.
Understanding of AI and programming by project members.
- As of March 2026, the adoption rate of BI-related AI was less than 10%.
- The market for BI projects is dominated by people born in the mid-1970s, who have relatively low understanding of new technologies such as AI.
- When AI is used in a project, there is a high probability that work will be concentrated on those who are performing AI-based tasks.
- As the project progresses and the AI proficiency of project members increases, it is expected that this combined with their existing domain knowledge will become a powerful weapon.
- Based on the above, it is anticipated that junior positions will become increasingly scarce.
Although the project is not yet complete and there are many challenges ahead, I feel like we have overcome a major hurdle.
The original plan was to use AI to carry out the project and introduce various AI skills and methodologies to improve our proficiency. However, due to being absorbed in the project work, we ended up improvising rather than following a proper PDCA or plan-based methodology.
It's a shame that this project is turning out to be less than ideal.
