Building AI Competence: A Self-Assessment Checklist

87% of lawyers now see technology adaptation as a key priority, yet many lack AI-specific training. State bars are introducing AI and technology CLE requirements. Some firms add AI knowledge as a requirement for partnership tracks. This self-assessment identifies gaps and provides a framework for development. ## Part 1: Technical Literacy **Foundation concepts**: Do you understand how large language models work, including basic architecture, training processes, and inference? Assessment questions: - [ ] Can you explain what "tokens" are and why they matter for AI input/output? - [ ] Do you understand the difference between a foundation model and a fine-tuned model? - [ ] Can you describe what RAG (Retrieval Augmented Generation) is and why legal AI tools use it? - [ ] Do you know what "temperature" settings control in AI output? - [ ] Can you explain why AI models have knowledge cutoff dates? **Limitations awareness**: Do you understand where AI systems fail? - [ ] Can you define "hallucination" in the AI context and explain why it occurs? - [ ] Do you understand "sycophancy" and how it manifests in legal research? - [ ] Can you identify situations where AI is likely to produce unreliable output? - [ ] Do you know what "misgrounding" means (correct statement, wrong citation)? - [ ] Can you explain why AI might perform differently on novel vs. common legal questions? **Risk assessment**: Can you evaluate AI appropriateness for specific tasks? - [ ] Can you categorize legal tasks by AI risk level (low/medium/high)? - [ ] Do you understand which matters should not use AI assistance? - [ ] Can you identify when confidentiality concerns prohibit AI use? - [ ] Do you know how to evaluate vendor data handling practices? ## Part 2: Verification Skills **Citation verification**: Can you confirm AI-generated legal citations? - [ ] Do you independently verify every case citation in AI output? - [ ] Do you confirm that cited cases actually support stated propositions? - [ ] Do you check that holdings are accurately characterized? - [ ] Do you verify quotations against original sources? - [ ] Do you confirm cases have not been overruled or distinguished? **Legal analysis verification**: Can you evaluate AI legal reasoning? - [ ] Can you identify logical gaps in AI-generated analysis? - [ ] Do you check AI output against your own understanding of the law? - [ ] Can you recognize when AI has missed relevant contrary authority? - [ ] Do you verify that AI correctly applies law to your specific facts? - [ ] Can you identify when AI has oversimplified nuanced legal questions? **Documentation**: Do you maintain verification records? - [ ] Do you document verification steps for AI-assisted work? - [ ] Can you demonstrate the verification process if challenged? - [ ] Do you maintain records of AI tools used and prompts provided? ## Part 3: Prompt Engineering **Effective prompting**: Can you craft prompts that produce reliable output? - [ ] Do you structure prompts with clear, specific instructions? - [ ] Do you provide relevant context to guide AI responses? - [ ] Do you know how to request citations and supporting authority? - [ ] Can you prompt for acknowledgment of uncertainty? - [ ] Do you understand how to avoid leading the AI toward desired conclusions? **Adversarial testing**: Do you challenge AI output? - [ ] Do you ask follow-up questions to test AI reasoning? - [ ] Do you prompt for contrary authority or counterarguments? - [ ] Do you test AI responses against known edge cases? - [ ] Do you verify AI does not simply agree with incorrect premises? ## Part 4: Ethical Compliance **Jurisdictional requirements**: Do you know applicable rules? - [ ] Have you reviewed AI guidance from your primary state bar? - [ ] Do you know whether your jurisdiction requires AI disclosure in filings? - [ ] Are you aware of CLE requirements for AI competence? - [ ] Do you understand multi-jurisdictional compliance obligations? **Confidentiality**: Do you protect client information? - [ ] Do you know how your AI tools handle client data? - [ ] Have you reviewed vendor agreements for data protection? - [ ] Do you avoid entering confidential information into non-secure tools? - [ ] Do you understand where AI processing occurs geographically? **Supervision**: Do you appropriately oversee AI use? - [ ] Do you treat AI vendors as supervised service providers? - [ ] Have you established firm policies for AI use? - [ ] Do you supervise staff AI use appropriately? - [ ] Do you review AI-assisted work before submission? ## Part 5: Ongoing Education **Current awareness**: Do you stay informed? - [ ] Do you track state bar AI guidance updates? - [ ] Do you follow legal AI developments and research? - [ ] Are you aware of the Stanford hallucination research findings? - [ ] Do you monitor AI-related ethics opinions and court decisions? **Formal education**: Have you pursued structured learning? - [ ] Have you completed AI-focused CLE programs? - [ ] Have you taken courses on legal technology or AI? - [ ] Do you participate in AI-focused professional development? - [ ] Have you obtained any AI competency certifications? **Practical development**: Are you building skills through use? - [ ] Do you regularly use AI tools in appropriate contexts? - [ ] Do you learn from verification failures when they occur? - [ ] Do you share learnings with colleagues? - [ ] Do you contribute to firm AI policies and training? ## Scoring and Development **Count your checked items:** - **0-15**: Significant gaps require immediate attention. Prioritize foundational education. - **16-30**: Developing competence. Focus on weak areas through targeted learning. - **31-40**: Strong foundation. Continue monitoring developments and refining skills. - **41-50**: Comprehensive competence. Consider training others and contributing to policy development. ## Development Resources **State bar resources**: Most bars now offer AI-specific CLE and guidance materials. **ABA resources**: Formal Opinion 512 and Task Force reports provide baseline understanding. **Academic programs**: Law schools including Case Western Reserve offer AI certification programs. **Industry training**: Legal AI vendors typically provide product-specific training. **Self-study**: Stanford HAI research, legal technology publications, and practitioner blogs offer ongoing education. ## The Competence Standard ABA Model Rule 1.1 requires competent representation, which Comment 8 specifies includes understanding "the benefits and risks associated with relevant technology." This is not about becoming a technologist. It is about understanding AI tools sufficiently to use them responsibly and to recognize when they fail. The checklist above operationalizes that obligation. Gaps identified here are not causes for alarm—they are targets for development. The lawyers who will thrive in AI-integrated practice are those who honestly assess their competence and systematically address deficiencies. --- ## Key Takeaways - Technical literacy includes understanding LLM basics, hallucination risks, and sycophancy - Verification skills require independent confirmation of citations, holdings, and legal analysis - Prompt engineering affects output quality; adversarial testing catches AI errors - Ethical compliance spans jurisdictional rules, confidentiality, and supervision obligations - Ongoing education is mandatory given rapid development in AI capabilities and requirements --- ## Sources **[Simbo AI: Maintaining Competence in the Age of AI - Essential Skills for Lawyers]** > Analysis of core competencies required for AI-assisted legal practice, including verification and ethical compliance. [Read Full Source →](https://www.simbo.ai/blog/maintaining-competence-in-the-age-of-ai-essential-skills-for-lawyers-to-effectively-utilize-artificial-intelligence-tools-2488208/) **[Law Tech AI: Future-Proofing Your Practice - 5 AI Skills Every Attorney Needs by 2026]** > Practical guide to skill development including prompt engineering and risk assessment capabilities. [Read Full Source →](https://law-tech.ai/future-proofing-your-practice-5-ai-skills-every-attorney-needs-by-2026/) **[UC IPCLJ: Training the Lawyers of Tomorrow - Why AI Literacy Must Be Integrated into Legal Education]** > Academic analysis of competency requirements and the gap between law school preparation and practice demands. [Read Full Source →](https://ucipclj.org/2025/11/12/training-the-lawyers-of-tomorrow-why-ai-literacy-must-be-integrated-into-legal-education/) **[NC Lawyers Weekly: Lawyers Face New Guidance on AI, Tech Competence]** > Coverage of state bar guidance developments and their implications for competency requirements. [Read Full Source →](https://nclawyersweekly.com/2025/09/29/lawyers-ai-ethics-competence-guidance/)