Predictive Science - AI Engineering & Prompt Architecture Lead - Vice President

Job Description

Take a lead role in acquiring, managing and retaining meaningful relationships that deliver outstanding experience to our customers. In this role, you will balance your focus on business results by offering options and finding solutions to help our customers with issues.

As a Vice President – AI Engineering & Prompt Architecture Lead in Predictive Science, you will own the technical vision and delivery for a team of prompt engineers and AI/ML model developers building intelligent document processing and data extraction solutions in the KYC/AML domain.

Job Responsibilities: 

• Architect agentic, multi-step AI workflows chaining classification → extraction → cross-validation → exception routing with human-in-the-loop checkpoints.
• Debug and remediate complex prompt failures (context-window overflow, instruction drift in long chains, RAG retrieval poisoning, output/format instability).
• Design prompt/model evaluation frameworks measuring accuracy plus consistency, robustness, latency, cost-per-call, and hallucination rate.
• Operationalize prompt lifecycle management as production code: versioning, CI/CD prompt tests, A/B experiments, rollback, and audited change history.
• Guide model selection and optimization (prompting vs fine-tuning vs custom training) balancing accuracy, latency, cost, and data sensitivity.
• Design RAG architectures for financial documents: chunking, embeddings, vector store design, re-ranking, and context injection.
• Oversee fine-tuning/training workflows: dataset curation, annotation quality, training configurations, and generalization across document variants.
• Build and maintain evaluation infrastructure: benchmark/golden datasets, regression suites, and automated scoring to catch regressions pre-production.
• Define confidence calibration and escalation logic so systems estimate uncertainty and route low-confidence outputs to human reviewers with the right context.
• Partner with governance/model risk and data engineering: produce validator-ready documentation (explainability/auditability) and ensure robust, refreshed data/annotation/eval pipelines; drive AI-native development practices to improve velocity.

Required qualifications, capabilities and skills:

• 10+ years in NLP/AI/ML or computational linguistics, including 3+ years leading technical teams with direct reports.
• Hands-on LLM internals expertise: tokenization impacts, attention limits, context window management, and temperature/sampling trade-offs.
• Proven prompt architecture design/debugging: multi-turn chains, few-/many-shot, chain-of-thought, self-consistency, and constitutional AI.
• Strong RAG system design experience: embeddings trade-offs (e.g., ada/BGE/Cohere), chunking for semi-structured docs, hybrid retrieval (dense+sparse), and re-ranking.
• Fine-tuning experience: LoRA/QLoRA, instruction tuning, RLHF/DPO, dataset curation, and evaluating tuned vs prompted performance.
• Python + ML engineering proficiency: PyTorch, Hugging Face, LangChain/LlamaIndex (or equivalents), vector DBs (Pinecone/Weaviate/pgvector), and API development.
• AI evaluation systems experience beyond F1: faithfulness, answer relevance, RAG context precision/recall, automated eval pipelines, and LLM-as-judge.
• Deep understanding of LLM failure modes: hallucinations, sycophancy, long-context instruction degradation, prompt-format sensitivity, and catastrophic forgetting.
• Structured output enforcement in production: JSON mode, function calling, constrained decoding, output parsers, and schema validation.
• Build-vs-buy/model selection track record: benchmarking foundation models (GPT-4/Claude/Llama/Mistral) against task requirements.
• Leadership under ambiguity: pragmatic trade-offs, rapid iteration as practices evolve, and strong communication with compliance, risk, and business stakeholders.

Preferred qualifications, capabilities and skills:

• Domain expertise in KYC/AML or financial document processing: entity extraction from registries, beneficial ownership structures, sanctions screening logic, adverse media classification.

• Experience designing autonomous AI agents: tool-use patterns, planning/reasoning loops, memory architectures, and safety guardrails for regulated environments.

• Knowledge of AI security/adversarial robustness: prompt injection defense, jailbreak detection, data poisoning awareness, and output monitoring for sensitive financial data.

• Experience with model distillation to produce smaller, faster models for cost-effective deployment.

• Familiarity with AI observability/monitoring: tracking prompt/model performance, drift detection, alerting, and health dashboards.

• Experience with multi-modal AI combining OCR, layout understanding, and LLM-based extraction for complex documents.

• Advanced degree (MS/PhD) in CS/NLP/ML or equivalent depth via publications, open-source, or production system design and Passion for talent development: growing engineers from junior prompt writers into senior AI system designers via structured mentorship.