Lowering AArch64 NEON `vfmaq_v` in Clang CIR

Acronyms and Terms

AArch64: the 64-bit Arm architecture.
NEON: Arm Advanced SIMD. SIMD means Single Instruction, Multiple Data.
NEON intrinsics: compiler-recognized C/C++ APIs that express Arm NEON operations, which the backend can lower to target instructions.
ACLE: Arm C Language Extensions, the C interface that exposes Arm architecture features such as NEON intrinsics.
AST: Abstract Syntax Tree, Clang’s structured representation of parsed and semantically checked source code.
IR: intermediate representation, a compiler-internal form between source code and machine code.
LLVM IR: (originally) Low-Level Virtual Machine IR, the main intermediate representation consumed by LLVM optimizers and backends.
CIR: Clang Intermediate Representation, also called ClangIR.
CIR-to-LLVM: the lowering path that translates CIR into LLVM IR.
CodeGen: Code Generation, the compiler stage that lowers checked source constructs into compiler IR.
FMA: fused multiply-add, a floating-point operation that computes a * b + c with one final rounding step.
FileCheck: the LLVM test tool used to match expected text patterns in compiler output.
MLIR: Multi-Level Intermediate Representation, the infrastructure used by CIR.
RUN lines: commands embedded in LLVM test files that tell the test runner how to compile the file and check the output.
builtin: a compiler-recognized operation. It may look like a function call in source or in a target header, but the compiler handles it specially instead of treating it as an ordinary library function call.
vfmaq: vector fused multiply-accumulate quad form. In ACLE source, it is written accumulator-first: vfmaq(a, b, c) means a + b * c.
lowering: translating something from a higher-level source or frontend representation into a lower-level IR that later compiler stages can optimize or turn into machine code.

High-Level Overview

This note is a write-up of LLVM commit cbf814d239b7a434d4a67606f5fbc4a234d0b51d. The commit teaches CIR CodeGen how to lower the NEON BI__builtin_neon_vfmaq_v builtin for AArch64. The useful part to write down is the path from source-level NEON intrinsic, through Clang’s builtin handling, to the final LLVM fma intrinsic, because that is what makes the patch more interesting than the number of changed lines suggests.

In this context, to “lower” means to translate a compiler-recognized operation into the IR form used by the next compiler stage. A “builtin” is the compiler-recognized operation itself: it may be reached through a normal-looking C wrapper, but Clang handles it directly rather than treating it as an ordinary library call.

In user-facing terms, this is the builtin used by ACLE wrappers such as vfmaq_f16, vfmaq_f32, and vfmaq_f64. The change is small in code, but it sits at an important point in Clang’s compilation pipeline: the point where frontend semantic information is translated into compiler IR.

A note on the commit’s stated scope: the upstream commit message is titled “Lower vfmaq_v f32/f64” and explicitly says vfmaq_f16 is “outside this PR”, yet the same diff adds a vfmaq_f16 test under +fullfp16. There is no contradiction in the actual implementation. All three wrappers — vfmaq_f16, vfmaq_f32, and vfmaq_f64 — lower through the same canonical builtin, BI__builtin_neon_vfmaq_v, so implementing that one CIR case enables all three at once. This post describes what the diff actually does, which is to cover f16, f32, and f64.

Clang CodeGen

Clang does not compile C or C++ source directly into machine code. The frontend first parses the source, performs semantic analysis, and builds an AST. After that, the CodeGen stage lowers the AST into an intermediate representation that later compiler stages can optimize and translate into target-specific code.

For the traditional Clang path, CodeGen emits LLVM IR directly. For example, a C function using a NEON intrinsic from arm_neon.h is parsed and type checked by the frontend, and then Clang CodeGen emits LLVM IR that represents the operation. LLVM’s optimizer and backend then lower that IR toward real AArch64 instructions.

Target builtins need special handling during CodeGen. A function such as vfmaq_f32(a, b, c) looks like a C function call in source, but it represents a specific NEON operation. The header maps it to a Clang builtin, and CodeGen is responsible for lowering that builtin to the correct IR operation.

At a high level, Clang treats a builtin as a function-like operation with a known compiler identity. The source may contain a normal-looking call, or a wrapper in arm_neon.h may call something like __builtin_neon_vfmaq_v. During parsing and semantic analysis, Clang recognizes that builtin name and records it as a call to a specific builtin ID rather than as an unresolved external function. Later, CodeGen switches on that builtin ID and emits the IR sequence that represents the operation for the target.

What CIR Adds

CIR is about giving Clang a middle layer between the AST and LLVM IR. Classic CodeGen lowers C/C++ semantics directly into LLVM IR, which is already fairly low-level. CIR is a newer CodeGen path that may eventually replace parts of that direct AST-to-LLVM IR path. It keeps more source-language meaning alive for longer, using MLIR infrastructure for verification, passes, and staged lowering, while making the lowering pipeline easier to inspect and evolve.

CIR is an MLIR-based intermediate representation emitted by Clang before lowering to LLVM IR. Instead of translating the AST directly to LLVM IR, the CIR path translates it to CIR first:

C/C++ source
  -> Clang AST
  -> CIR
  -> LLVM IR
  -> LLVM optimization and backend
  -> machine code

This makes CIR-to-LLVM the bridge from ClangIR into the existing LLVM pipeline. If a builtin is not implemented in CIR CodeGen, compiling with -fclangir can fail even when normal LLVM IR CodeGen already supports the same source program.

That is the gap this commit addresses for BI__builtin_neon_vfmaq_v.

CodeGen Tests Under `clang/test/CodeGen/AArch64/`

The tests under clang/test/CodeGen/AArch64/ are regression tests for the frontend CodeGen stage. They compile small C functions and use FileCheck to verify the generated IR.

Historically, many AArch64 NEON tests lived in broad files such as:

clang/test/CodeGen/AArch64/neon-intrinsics.c
clang/test/CodeGen/AArch64/v8.2a-neon-intrinsics.c

Those files mainly check the traditional direct-to-LLVM CodeGen path.

The CIR-enabled tests being used here live under:

clang/test/CodeGen/AArch64/neon/

These files are organized by intrinsic family, and the relevant tests include RUN lines for both direct LLVM CodeGen and ClangIR. A typical CIR-enabled test can check three related outputs:

direct LLVM IR CodeGen, without -fclangir
CIR-to-LLVM output, with -fclangir -emit-llvm
raw CIR output, with -fclangir -emit-cir

This is useful because the direct LLVM path is the historical behavior, while the CIR paths prove that ClangIR emits equivalent semantics and also produces the expected CIR operations.

In this commit, the existing vfmaq coverage was moved into CIR-enabled test files:

vfmaq_f32 and vfmaq_f64 moved into clang/test/CodeGen/AArch64/neon/fused-multiply.c
vfmaq_f16 moved into clang/test/CodeGen/AArch64/neon/fused-multiple-fullfp16.c (the filename uses “multiple” upstream, which is a typo for “multiply”; the blog reproduces the exact filename so paths match the tree)

The old checks were preserved for LLVM IR, and new CIR checks were added. The move is not just file cleanup: it lets one focused test file verify both the existing direct LLVM CodeGen behavior and the new CIR implementation.

Detailed Walkthrough

What `vfmaq_v` Represents

The ACLE vfmaq_* intrinsics are vector fused multiply-accumulate operations. At a high level:

vfmaq(a, b, c)

means:

a + b * c

with fused floating-point semantics. “Fused” means the multiply and add are treated as one operation for rounding purposes, rather than rounding after the multiply and then rounding again after the add.

The q in vfmaq means this is the quad-vector form. The covered wrappers operate on these vector shapes:

vfmaq_f16: 8 lanes of half precision, represented in LLVM IR as <8 x half>
vfmaq_f32: 4 lanes of single precision, represented as <4 x float>
vfmaq_f64: 2 lanes of double precision, represented as <2 x double>

Although the C wrappers have different element types, they lower through the same canonical Clang builtin: BI__builtin_neon_vfmaq_v.

Operand Order

The NEON intrinsic and the LLVM intrinsic express the same mathematical operation, but their operand order is different.

The NEON wrapper is accumulator-first:

vfmaq(a, b, c)  // a + b * c

This is not just an interpretation of the ACLE name. The existing traditional CodeGen path has an explicit comment for the same builtin in clang/lib/CodeGen/TargetBuiltins/ARM.cpp:

// NEON intrinsic puts accumulator first, unlike the LLVM fma.
return emitCallMaybeConstrainedFPBuiltin(
    *this, Intrinsic::fma, Intrinsic::experimental_constrained_fma, Ty,
    {Ops[1], Ops[2], Ops[0]});

That code handles BI__builtin_neon_vfmaq_v by passing operand 0 last to LLVM fma, which identifies operand 0 as the accumulator. The generated arm_neon.h wrapper supports the same reading: vfmaq_f32(__p0, __p1, __p2) forwards __p0, __p1, and __p2 to __builtin_neon_vfmaq_v in that order.

In arm_neon.td, this builtin is generated by three SInst<"vfma", "....", X> definitions, one per element type:

def VFMA   : SInst<"vfma", "....", "fQf">;  // f32, single + quad
def FMLA   : SInst<"vfma", "....", "dQd">;  // f64, scalar + quad (AArch64)
def VFMAH  : SInst<"vfma", "....", "hQh">;  // f16, single + quad (+fullfp16)

The third string is a list of type modifiers. Lower-case f/d/h is the non-q form, and the Q-prefixed variant is the quad form, which is what generates vfmaq_f32, vfmaq_f64, and vfmaq_f16. All three feed the same BI__builtin_neon_vfmaq_v builtin, which is why one CIR case is enough to cover all three wrappers.

LLVM’s fma intrinsic is multiply-operands-first:

llvm.fma(b, c, a)  ; b * c + a

So the lowering cannot simply forward the operands in source order. It must reorder them.

CIR Lowering

Before this commit, both vfma_v and vfmaq_v shared the same fall-through in emitCommonNeonBuiltinExpr and ended up on a generic unimplemented path, which is why building any user code that called these wrappers with -fclangir failed. The relevant pre-patch fragment looked like this:

case NEON::BI__builtin_neon_vfma_v:
case NEON::BI__builtin_neon_vfmaq_v:
  // ... fell through with other unimplemented cases ...

The patch splits these two cases apart. vfma_v becomes an explicit errorNYI, which keeps the diagnostic clean for the form that is still unimplemented, while vfmaq_v gets a dedicated implementation:

case NEON::BI__builtin_neon_vfma_v:
  cgf.cgm.errorNYI(expr->getSourceRange(),
                   std::string("unimplemented AArch64 builtin call: ") +
                       ctx.BuiltinInfo.getName(builtinID));
  return mlir::Value{};
case NEON::BI__builtin_neon_vfmaq_v: {
  // ... new implementation, shown below ...
}

The new vfmaq_v implementation bitcasts all three operands to the expected result vector type and then emits the LLVM fma intrinsic with reordered operands:

mlir::Value op0 = cgf.getBuilder().createBitcast(ops[0], ty);
mlir::Value op1 = cgf.getBuilder().createBitcast(ops[1], ty);
mlir::Value op2 = cgf.getBuilder().createBitcast(ops[2], ty);
llvm::SmallVector<mlir::Value> fmaOps = {op1, op2, op0};
return emitCallMaybeConstrainedBuiltin(cgf.getBuilder(), loc, "fma", ty,
                                       fmaOps);

Here:

ops[0] is the accumulator a
ops[1] is the first multiplicand b
ops[2] is the second multiplicand c
the emitted intrinsic receives {b, c, a}

That preserves the source-level NEON semantics while using LLVM’s canonical fma operation.

Why the Bitcasts Are There

The tests show several bitcasts before the final llvm.fma.* call. This is normal for NEON CodeGen. NEON vector values often pass through canonical integer or byte-vector forms while Clang normalizes builtin operands. Before calling the floating-point LLVM intrinsic, CodeGen must recover the expected floating-point vector type.

For example, vfmaq_f32 ultimately emits an LLVM intrinsic of this shape:

call <4 x float> @llvm.fma.v4f32(...)

Similarly:

call <8 x half>   @llvm.fma.v8f16(...)
call <2 x double> @llvm.fma.v2f64(...)

The important semantic check is that the final fma call receives the multiplicands first and the accumulator last.

How the Tests Prove the Change

The moved tests prove three things at once, one per RUN line:

the existing direct LLVM CodeGen behavior remains unchanged — checked by the first RUN line, without -fclangir, against the LLVM prefix
CIR-to-LLVM produces matching LLVM IR — checked by the second RUN line, with -fclangir -emit-llvm, against the same LLVM prefix (so the same check lines must hold for both pipelines)
the new CIR implementation emits the expected CIR operation — checked by the third RUN line, with -fclangir -emit-cir, against the CIR prefix

For direct LLVM CodeGen and CIR-to-LLVM, the tests check the final LLVM intrinsic call. For raw CIR output, they check the CIR intrinsic operation. A representative CIR check looks like this:

// CIR: cir.call_llvm_intrinsic "fma" %, %, % :

A representative LLVM check looks for the typed LLVM intrinsic:

// LLVM-NEXT: [[FMA:%.*]] = call <4 x float> @llvm.fma.v4f32(...)

That combination catches both sides of the implementation: whether CIR can represent the builtin, and whether CIR-to-LLVM produces the same kind of LLVM IR as the traditional CodeGen path.

From LLVM IR to AArch64 Assembly

The commit stops at LLVM IR tests, but that IR is not the end of the compiler pipeline. LLVM IR is still target-independent enough that it does not name a specific AArch64 instruction. The AArch64 backend is responsible for turning it into real ARM64 assembly.

For the vfmaq_f32 case moved into clang/test/CodeGen/AArch64/neon/fused-multiply.c, the important LLVM check from the commit is:

// LLVM-NEXT: [[FMA:%.*]] = call <4 x float> @llvm.fma.v4f32(
// LLVM-SAME:     <4 x float> [[B_CAST]],
// LLVM-SAME:     <4 x float> [[C_CAST]],
// LLVM-SAME:     <4 x float> [[A_CAST]])

The exact FileCheck formatting in the test is more compact, but the important semantic point is this:

call <4 x float> @llvm.fma.v4f32(b, c, a)

This LLVM intrinsic means:

b * c + a

That is the same operation as:

vfmaq_f32(a, b, c)

because the NEON wrapper is accumulator-first while LLVM fma is multiply-operands-first.

Backend Lowering

After Clang emits LLVM IR, the LLVM backend takes over. In broad strokes, the remaining path is:

LLVM IR
  -> LLVM optimization passes
  -> AArch64 instruction selection
  -> register allocation
  -> assembly printing
  -> assembler/object code

Instruction selection is the stage that recognizes an operation such as:

%f = call <4 x float> @llvm.fma.v4f32(<4 x float> %b,
                                      <4 x float> %c,
                                      <4 x float> %a)

and chooses an AArch64 instruction that implements it. For four single-precision floating-point lanes, the natural AArch64 NEON instruction is fmla on a .4s vector:

fmla v0.4s, v2.4s, v1.4s

An “explain it like I’m five” version is:

LLVM IR says: "do a four-lane fused multiply-add."
The AArch64 backend says: "on this machine, that is an fmla instruction."

Why the Assembly Looks Slightly Different

At first glance, the assembly can look like it changed the operand order again. For a small standalone LLVM IR function like this:

define <4 x float> @test(<4 x float> %a,
                         <4 x float> %b,
                         <4 x float> %c) {
entry:
  %f = call <4 x float> @llvm.fma.v4f32(<4 x float> %b,
                                        <4 x float> %c,
                                        <4 x float> %a)
  ret <4 x float> %f
}

llc can produce:

test:
  fmla v0.4s, v2.4s, v1.4s
  ret

The reason this is still correct is the AArch64 calling convention and the shape of the fmla instruction.

For this function, the incoming vector arguments are naturally assigned to vector registers:

%a -> v0
%b -> v1
%c -> v2

The AArch64 fmla vector instruction updates its destination register:

Vd = Vd + Vn * Vm

So:

fmla v0.4s, v2.4s, v1.4s

means:

v0 = v0 + v2 * v1

The backend has placed the two multiplicand operands in the two source slots of the fmla instruction. The important correctness requirement is that the accumulator/addend is still v0, which is the original %a, and the result is also returned in v0.

That gives the same source-level meaning:

a + b * c

This is the connection between the commit and the final machine instruction: the CIR lowering added in CIRGenBuiltinAArch64.cpp makes sure the LLVM IR has the canonical llvm.fma.v4f32(b, c, a) form. Once that IR exists, the AArch64 backend can select the actual NEON fmla SIMD instruction.

Code References

The main implementation reference is the new BI__builtin_neon_vfmaq_v case in clang/lib/CIR/CodeGen/CIRGenBuiltinAArch64.cpp, already shown in full under “CIR Lowering” above. It bitcasts the three operands to the result vector type, builds fmaOps = {op1, op2, op0}, and emits the shared "fma" intrinsic via emitCallMaybeConstrainedBuiltin.

The test reference is clang/test/CodeGen/AArch64/neon/fused-multiply.c. The vfmaq_f32 test calls the source-level wrapper:

return vfmaq_f32(a, b, c);

and checks that LLVM IR receives the operands as:

@llvm.fma.v4f32([[B_CAST]], [[C_CAST]], [[A_CAST]])

The classic CodeGen reference is clang/lib/CodeGen/TargetBuiltins/ARM.cpp, which already had the same operand rule for the direct LLVM CodeGen path:

// NEON intrinsic puts accumulator first, unlike the LLVM fma.
return emitCallMaybeConstrainedFPBuiltin(
    *this, Intrinsic::fma, Intrinsic::experimental_constrained_fma, Ty,
    {Ops[1], Ops[2], Ops[0]});

Summary

The commit adds CIR support for the canonical AArch64 NEON BI__builtin_neon_vfmaq_v builtin. That enables the ClangIR path for the vfmaq_f16, vfmaq_f32, and vfmaq_f64 vector quad fused multiply-accumulate wrappers.

The central point is operand mapping: NEON spells the operation as accumulator-first, while LLVM’s fma intrinsic expects the two multiplicands first and the accumulator last. The lowering performs that reordering and emits the shared fma intrinsic, while the moved tests verify the result through both direct LLVM CodeGen and CIR-based CodeGen.

Lowering AArch64 NEON vfmaq_v in Clang CIR