[MCA] Adding the CustomBehaviour class to llvm-mca

Some instructions are not defined well enough within the target’s scheduling
model for llvm-mca to be able to properly simulate its behaviour. The ideal
solution to this situation is to modify the scheduling model, but that’s not
always a viable strategy. Maybe other parts of the backend depend on that
instruction being modelled the way that it is. Or maybe the instruction is quite
complex and it’s difficult to fully capture its behaviour with tablegen. The
CustomBehaviour class (which I will refer to as CB frequently) is designed to
provide intuitive scaffolding for developers to implement the correct modelling
for these instructions.

Implementation details:

llvm-mca does its best to extract relevant register, resource, and memory
information from every MCInst when lowering them to an mca::Instruction. It then
uses this information to detect dependencies and simulate stalls within the
pipeline. For some instructions, the information that gets captured within the
mca::Instruction is not enough for mca to simulate them properly. In these
cases, there are two main possibilities:

1. The instruction has a dependency that isn’t detected by mca.
2. mca is incorrectly enforcing a dependency that shouldn’t exist.

For the rest of this discussion, I will be focusing on (1), but I have put some
thought into (2) and I may revisit it in the future.

So we have an instruction that has dependencies that aren’t picked up by mca.
The basic idea for both pipelines in mca is that when an instruction wants to be
dispatched, we first check for register hazards and then we check for resource
hazards. This is where CB is injected. If no register or resource hazards have
been detected, we make a call to CustomBehaviour::checkCustomHazard() to give
the target specific CB the chance to detect and enforce any custom dependencies.

The return value for checkCustomHazaard() is an unsigned int representing the
(minimum) number of cycles that the instruction needs to stall for. It’s fine to
underestimate this value because when StallCycles gets down to 0, we’ll end up
checking for all the hazards again before the instruction is actually
dispatched. However, it’s important not to overestimate the value and the more
accurate your estimate is, the more efficient mca’s execution can be.

In general, for checkCustomHazard() to be able to detect these custom
dependencies, it needs information about the current instruction and also all of
the instructions that are still executing within the pipeline. The mca pipeline
uses mca::Instruction rather than MCInst and the current information encoded
within each mca::Instruction isn’t sufficient for my use cases. I had to add a
few extra attributes to the mca::Instruction class and have them get set by the
MCInst during instruction building. For example, the current mca::Instruction
doesn’t know its opcode, and it also doesn’t know anything about its immediate
operands (both of which I had to add to the class).

With information about the current instruction, a list of all currently
executing instructions, and some target specific objects (MCSubtargetInfo and
MCInstrInfo which the base CB class has references to), developers should be
able to detect and enforce most custom dependencies within checkCustomHazard. If
you need more information than is present in the mca::Instruction, feel free to
add attributes to that class and have them set during the lowering sequence from
MCInst.

Fortunately, in the in-order pipeline, it’s very convenient for us to pass these
arguments to checkCustomHazard. The hazard checking is taken care of within
InOrderIssueStage::canExecute(). This function takes a const InstRef as a
parameter (representing the instruction that currently wants to be dispatched)
and the InOrderIssueStage class maintains a SmallVector<InstRef, 4> which holds
all of the currently executing instructions. For the out-of-order pipeline, it’s
a bit trickier to get the list of executing instructions and this is why I have
held off on implementing it myself. This is the main topic I will bring up when
I eventually make a post to discuss and ask for feedback.

CB is a base class where targets implement their own derived classes. If a
target specific CB does not exist (or we pass in the -disable-cb flag), the base
class is used. This base class trivially returns 0 from its checkCustomHazard()
implementation (meaning that the current instruction needs to stall for 0 cycles
aka no hazard is detected). For this reason, targets or users who choose not to
use CB shouldn’t see any negative impacts to accuracy or performance (in
comparison to pre-patch llvm-mca).

Differential Revision: https://reviews.llvm.org/D104149

18 files changed, 488 insertions, 18 deletions

diff --git a/llvm/docs/CommandGuide/llvm-mca.rst b/llvm/docs/CommandGuide/llvm-mca.rst
index 9e40e5d..abb932e 100644
--- a/llvm/docs/CommandGuide/llvm-mca.rst
+++ b/llvm/docs/CommandGuide/llvm-mca.rst
@@ -212,6 +212,11 @@ option specifies "``-``", then the output will also be sent to standard output.
   Print the requested views in JSON format. The instructions and the processor
   resources are printed as members of special top level JSON objects.  The
   individual views refer to them by index.
+  
+.. option:: -disable-cb
+
+  Force usage of the generic CustomBehaviour class rather than using the target
+  specific class. The generic class never detects any custom hazards.
 
 
 EXIT STATUS
@@ -978,3 +983,32 @@ Once issued, an instruction is moved to ``IssuedInst`` set until it is ready to
 retire. :program:`llvm-mca` ensures that writes are committed in-order. However,
 an instruction is allowed to commit writes and retire out-of-order if
 ``RetireOOO`` property is true for at least one of its writes.
+
+Custom Behaviour
+""""""""""""""""""""""""""""""""""""
+Due to certain instructions not being expressed perfectly within their
+scheduling model, :program:`llvm-ma` isn't always able to simulate them
+perfectly. Modifying the scheduling model isn't always a viable
+option though (maybe because the instruction is modeled incorrectly on
+purpose or the instruction's behaviour is quite complex). The
+CustomBehaviour class can be used in these cases to enforce proper
+instruction modeling (often by customizing data dependencies and detecting
+hazards that :program:`llvm-ma` has no way of knowing about).
+
+:program:`llvm-mca` comes with one generic and multiple target specific
+CustomBehaviour classes. The generic class will be used if the ``-disable-cb``
+flag is used or if a target specific CustomBehaviour class doesn't exist for
+that target. (The generic class does nothing.) Currently, the CustomBehaviour
+class is only a part of the in-order pipeline, but there are plans to add it
+to the out-of-order pipeline in the future.
+
+CustomBehaviour's main method is `checkCustomHazard()` which uses the
+current instruction and a list of all instructions still executing within
+the pipeline to determine if the current instruction should be dispatched.
+As output, the method returns an integer representing the number of cycles
+that the current instruction must stall for (this can be an underestimate
+if you don't know the exact number and a value of 0 represents no stall).
+
+If you'd like to add a CustomBehaviour class for a target that doesn't
+already have one, refer to an existing implementation to see how to set it
+up. Remember to look at (and add to) `/llvm-mca/lib/CMakeLists.txt`.
diff --git a/llvm/include/llvm/MCA/Context.h b/llvm/include/llvm/MCA/Context.h
index 23591c1..0abcfd7 100644
--- a/llvm/include/llvm/MCA/Context.h
+++ b/llvm/include/llvm/MCA/Context.h
@@ -19,6 +19,7 @@
 
 #include "llvm/MC/MCRegisterInfo.h"
 #include "llvm/MC/MCSubtargetInfo.h"
+#include "llvm/MCA/CustomBehaviour.h"
 #include "llvm/MCA/HardwareUnits/HardwareUnit.h"
 #include "llvm/MCA/Pipeline.h"
 #include "llvm/MCA/SourceMgr.h"
@@ -67,12 +68,14 @@ public:
   /// Construct a basic pipeline for simulating an out-of-order pipeline.
   /// This pipeline consists of Fetch, Dispatch, Execute, and Retire stages.
   std::unique_ptr<Pipeline> createDefaultPipeline(const PipelineOptions &Opts,
-                                                  SourceMgr &SrcMgr);
+                                                  SourceMgr &SrcMgr,
+                                                  CustomBehaviour &CB);
 
   /// Construct a basic pipeline for simulating an in-order pipeline.
   /// This pipeline consists of Fetch, InOrderIssue, and Retire stages.
   std::unique_ptr<Pipeline> createInOrderPipeline(const PipelineOptions &Opts,
-                                                  SourceMgr &SrcMgr);
+                                                  SourceMgr &SrcMgr,
+                                                  CustomBehaviour &CB);
 };
 
 } // namespace mca
diff --git a/llvm/include/llvm/MCA/CustomBehaviour.h b/llvm/include/llvm/MCA/CustomBehaviour.h
new file mode 100644
index 0000000..246a5de
--- /dev/null
+++ b/llvm/include/llvm/MCA/CustomBehaviour.h
@@ -0,0 +1,86 @@
+//===---------------------- CustomBehaviour.h -------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+/// \file
+///
+/// This file defines the base class CustomBehaviour which can be inherited from
+/// by specific targets (ex. llvm/tools/llvm-mca/lib/X86CustomBehaviour.h).
+/// CustomBehaviour is designed to enforce custom behaviour and dependencies
+/// within the llvm-mca pipeline simulation that llvm-mca isn't already capable
+/// of extracting from the Scheduling Models.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_MCA_CUSTOMBEHAVIOUR_H
+#define LLVM_MCA_CUSTOMBEHAVIOUR_H
+
+#include "llvm/MC/MCInst.h"
+#include "llvm/MC/MCInstrInfo.h"
+#include "llvm/MC/MCSubtargetInfo.h"
+#include "llvm/MCA/SourceMgr.h"
+
+namespace llvm {
+namespace mca {
+
+/// Class which can be overriden by targets to modify the
+/// mca::Instruction objects before the pipeline starts.
+/// A common usage of this class is to add immediate operands to certain
+/// instructions or to remove Defs/Uses from an instruction where the
+/// schedulinng model is incorrect.
+class InstrPostProcess {
+protected:
+  const MCSubtargetInfo &STI;
+  const MCInstrInfo &MCII;
+
+public:
+  InstrPostProcess(const MCSubtargetInfo &STI, const MCInstrInfo &MCII)
+      : STI(STI), MCII(MCII) {}
+
+  virtual ~InstrPostProcess() {}
+
+  virtual void postProcessInstruction(std::unique_ptr<Instruction> &Inst,
+                                      const MCInst &MCI) {}
+};
+
+/// Class which can be overriden by targets to enforce instruction
+/// dependencies and behaviours that aren't expressed well enough
+/// within the scheduling model for mca to automatically simulate
+/// them properly.
+/// If you implement this class for your target, make sure to also implement
+/// a target specific InstrPostProcess class as well.
+class CustomBehaviour {
+protected:
+  const MCSubtargetInfo &STI;
+  const SourceMgr &SrcMgr;
+  const MCInstrInfo &MCII;
+
+public:
+  CustomBehaviour(const MCSubtargetInfo &STI, const SourceMgr &SrcMgr,
+                  const MCInstrInfo &MCII)
+      : STI(STI), SrcMgr(SrcMgr), MCII(MCII) {}
+
+  virtual ~CustomBehaviour() {}
+
+  // Before the llvm-mca pipeline dispatches an instruction, it first checks
+  // for any register or resource dependencies / hazards. If it doesn't find
+  // any, this method will be invoked to determine if there are any custom
+  // hazards that the instruction needs to wait for.
+  // The return value of this method is the number of cycles that the
+  // instruction needs to wait for.
+  // It's safe to underestimate the number of cycles to wait for since these
+  // checks will be invoked again before the intruction gets dispatched.
+  // However, it's not safe (accurate) to overestimate the number of cycles
+  // to wait for since the instruction will wait for AT LEAST that number of
+  // cycles before attempting to be dispatched again.
+  virtual unsigned checkCustomHazard(ArrayRef<InstRef> IssuedInst,
+                                     const InstRef &IR);
+};
+
+} // namespace mca
+} // namespace llvm
+
+#endif /* LLVM_MCA_CUSTOMBEHAVIOUR_H */
diff --git a/llvm/include/llvm/MCA/HWEventListener.h b/llvm/include/llvm/MCA/HWEventListener.h
index aeb5ab1..5b5b83c 100644
--- a/llvm/include/llvm/MCA/HWEventListener.h
+++ b/llvm/include/llvm/MCA/HWEventListener.h
@@ -115,6 +115,7 @@ public:
     SchedulerQueueFull,
     LoadQueueFull,
     StoreQueueFull,
+    CustomBehaviourStall,
     LastGenericEvent
   };
 
diff --git a/llvm/include/llvm/MCA/Instruction.h b/llvm/include/llvm/MCA/Instruction.h
index f34f31d..bb837a2 100644
--- a/llvm/include/llvm/MCA/Instruction.h
+++ b/llvm/include/llvm/MCA/Instruction.h
@@ -33,6 +33,104 @@ namespace mca {
 
 constexpr int UNKNOWN_CYCLES = -512;
 
+/// A representation of an mca::Instruction operand
+/// for use in mca::CustomBehaviour.
+class MCAOperand {
+  // This class is mostly copied from MCOperand within
+  // MCInst.h except that we don't keep track of
+  // expressions or sub-instructions.
+  enum MCAOperandType : unsigned char {
+    kInvalid,   ///< Uninitialized, Relocatable immediate, or Sub-instruction.
+    kRegister,  ///< Register operand.
+    kImmediate, ///< Immediate operand.
+    kSFPImmediate, ///< Single-floating-point immediate operand.
+    kDFPImmediate, ///< Double-Floating-point immediate operand.
+  };
+  MCAOperandType Kind = kInvalid;
+
+  union {
+    unsigned RegVal;
+    int64_t ImmVal;
+    uint32_t SFPImmVal;
+    uint64_t FPImmVal;
+  };
+
+  // We only store specific operands for specific instructions
+  // so an instruction's operand 3 may be stored within the list
+  // of MCAOperand as element 0. This Index attribute keeps track
+  // of the original index (3 for this example).
+  unsigned Index;
+
+public:
+  MCAOperand() : FPImmVal(0) {}
+
+  bool isValid() const { return Kind != kInvalid; }
+  bool isReg() const { return Kind == kRegister; }
+  bool isImm() const { return Kind == kImmediate; }
+  bool isSFPImm() const { return Kind == kSFPImmediate; }
+  bool isDFPImm() const { return Kind == kDFPImmediate; }
+
+  /// Returns the register number.
+  unsigned getReg() const {
+    assert(isReg() && "This is not a register operand!");
+    return RegVal;
+  }
+
+  int64_t getImm() const {
+    assert(isImm() && "This is not an immediate");
+    return ImmVal;
+  }
+
+  uint32_t getSFPImm() const {
+    assert(isSFPImm() && "This is not an SFP immediate");
+    return SFPImmVal;
+  }
+
+  uint64_t getDFPImm() const {
+    assert(isDFPImm() && "This is not an FP immediate");
+    return FPImmVal;
+  }
+
+  void setIndex(const unsigned Idx) { Index = Idx; }
+
+  unsigned getIndex() const { return Index; }
+
+  static MCAOperand createReg(unsigned Reg) {
+    MCAOperand Op;
+    Op.Kind = kRegister;
+    Op.RegVal = Reg;
+    return Op;
+  }
+
+  static MCAOperand createImm(int64_t Val) {
+    MCAOperand Op;
+    Op.Kind = kImmediate;
+    Op.ImmVal = Val;
+    return Op;
+  }
+
+  static MCAOperand createSFPImm(uint32_t Val) {
+    MCAOperand Op;
+    Op.Kind = kSFPImmediate;
+    Op.SFPImmVal = Val;
+    return Op;
+  }
+
+  static MCAOperand createDFPImm(uint64_t Val) {
+    MCAOperand Op;
+    Op.Kind = kDFPImmediate;
+    Op.FPImmVal = Val;
+    return Op;
+  }
+
+  static MCAOperand createInvalid() {
+    MCAOperand Op;
+    Op.Kind = kInvalid;
+    Op.FPImmVal = 0;
+    return Op;
+  }
+};
+
 /// A register write descriptor.
 struct WriteDescriptor {
   // Operand index. The index is negative for implicit writes only.
@@ -160,6 +258,7 @@ public:
   int getCyclesLeft() const { return CyclesLeft; }
   unsigned getWriteResourceID() const { return WD->SClassOrWriteResourceID; }
   MCPhysReg getRegisterID() const { return RegisterID; }
+  void setRegisterID(const MCPhysReg RegID) { RegisterID = RegID; }
   unsigned getRegisterFileID() const { return PRFID; }
   unsigned getLatency() const { return WD->Latency; }
   unsigned getDependentWriteCyclesLeft() const {
@@ -409,8 +508,15 @@ class InstructionBase {
   // One entry per each implicit and explicit register use.
   SmallVector<ReadState, 4> Uses;
 
+  // List of operands which can be used by mca::CustomBehaviour
+  std::vector<MCAOperand> Operands;
+
+  // Instruction opcode which can be used by mca::CustomBehaviour
+  unsigned Opcode;
+
 public:
-  InstructionBase(const InstrDesc &D) : Desc(D), IsOptimizableMove(false) {}
+  InstructionBase(const InstrDesc &D, const unsigned Opcode)
+      : Desc(D), IsOptimizableMove(false), Operands(0), Opcode(Opcode) {}
 
   SmallVectorImpl<WriteState> &getDefs() { return Defs; }
   ArrayRef<WriteState> getDefs() const { return Defs; }
@@ -420,6 +526,20 @@ public:
 
   unsigned getLatency() const { return Desc.MaxLatency; }
   unsigned getNumMicroOps() const { return Desc.NumMicroOps; }
+  unsigned getOpcode() const { return Opcode; }
+
+  /// Return the MCAOperand which corresponds to index Idx within the original
+  /// MCInst.
+  const MCAOperand *getOperand(const unsigned Idx) const {
+    auto It = std::find_if(
+        Operands.begin(), Operands.end(),
+        [&Idx](const MCAOperand &Op) { return Op.getIndex() == Idx; });
+    if (It == Operands.end())
+      return nullptr;
+    return &(*It);
+  }
+  unsigned getNumOperands() const { return Operands.size(); }
+  void addOperand(const MCAOperand Op) { Operands.push_back(Op); }
 
   bool hasDependentUsers() const {
     return any_of(Defs,
@@ -490,11 +610,11 @@ class Instruction : public InstructionBase {
   bool IsEliminated;
 
 public:
-  Instruction(const InstrDesc &D)
-      : InstructionBase(D), Stage(IS_INVALID), CyclesLeft(UNKNOWN_CYCLES),
-        RCUTokenID(0), LSUTokenID(0), UsedBuffers(D.UsedBuffers),
-        CriticalRegDep(), CriticalMemDep(), CriticalResourceMask(0),
-        IsEliminated(false) {}
+  Instruction(const InstrDesc &D, const unsigned Opcode)
+      : InstructionBase(D, Opcode), Stage(IS_INVALID),
+        CyclesLeft(UNKNOWN_CYCLES), RCUTokenID(0), LSUTokenID(0),
+        UsedBuffers(D.UsedBuffers), CriticalRegDep(), CriticalMemDep(),
+        CriticalResourceMask(0), IsEliminated(false) {}
 
   unsigned getRCUTokenID() const { return RCUTokenID; }
   unsigned getLSUTokenID() const { return LSUTokenID; }
diff --git a/llvm/include/llvm/MCA/Stages/InOrderIssueStage.h b/llvm/include/llvm/MCA/Stages/InOrderIssueStage.h
index 1c1f91e..b7006e7 100644
--- a/llvm/include/llvm/MCA/Stages/InOrderIssueStage.h
+++ b/llvm/include/llvm/MCA/Stages/InOrderIssueStage.h
@@ -14,6 +14,7 @@
 #ifndef LLVM_MCA_STAGES_INORDERISSUESTAGE_H
 #define LLVM_MCA_STAGES_INORDERISSUESTAGE_H
 
+#include "llvm/MCA/CustomBehaviour.h"
 #include "llvm/MCA/HardwareUnits/ResourceManager.h"
 #include "llvm/MCA/SourceMgr.h"
 #include "llvm/MCA/Stages/Stage.h"
@@ -23,7 +24,13 @@ namespace mca {
 class RegisterFile;
 
 struct StallInfo {
-  enum class StallKind { DEFAULT, REGISTER_DEPS, DISPATCH, DELAY };
+  enum class StallKind {
+    DEFAULT,
+    REGISTER_DEPS,
+    DISPATCH,
+    DELAY,
+    CUSTOM_STALL
+  };
 
   InstRef IR;
   unsigned CyclesLeft;
@@ -46,6 +53,7 @@ class InOrderIssueStage final : public Stage {
   const MCSubtargetInfo &STI;
   RegisterFile &PRF;
   ResourceManager RM;
+  CustomBehaviour &CB;
 
   /// Instructions that were issued, but not executed yet.
   SmallVector<InstRef, 4> IssuedInst;
@@ -101,7 +109,8 @@ class InOrderIssueStage final : public Stage {
   void retireInstruction(InstRef &IR);
 
 public:
-  InOrderIssueStage(const MCSubtargetInfo &STI, RegisterFile &PRF);
+  InOrderIssueStage(const MCSubtargetInfo &STI, RegisterFile &PRF,
+                    CustomBehaviour &CB);
 
   unsigned getIssueWidth() const;
   bool isAvailable(const InstRef &) const override;
diff --git a/llvm/lib/MCA/CMakeLists.txt b/llvm/lib/MCA/CMakeLists.txt
index 6ade8ba..38156be 100644
--- a/llvm/lib/MCA/CMakeLists.txt
+++ b/llvm/lib/MCA/CMakeLists.txt
@@ -1,6 +1,7 @@
 add_llvm_component_library(LLVMMCA
   CodeEmitter.cpp
   Context.cpp
+  CustomBehaviour.cpp
   HWEventListener.cpp
   HardwareUnits/HardwareUnit.cpp
   HardwareUnits/LSUnit.cpp
diff --git a/llvm/lib/MCA/Context.cpp b/llvm/lib/MCA/Context.cpp
index 1433448..99d2373 100644
--- a/llvm/lib/MCA/Context.cpp
+++ b/llvm/lib/MCA/Context.cpp
@@ -29,11 +29,12 @@ namespace llvm {
 namespace mca {
 
 std::unique_ptr<Pipeline>
-Context::createDefaultPipeline(const PipelineOptions &Opts, SourceMgr &SrcMgr) {
+Context::createDefaultPipeline(const PipelineOptions &Opts, SourceMgr &SrcMgr,
+                               CustomBehaviour &CB) {
   const MCSchedModel &SM = STI.getSchedModel();
 
   if (!SM.isOutOfOrder())
-    return createInOrderPipeline(Opts, SrcMgr);
+    return createInOrderPipeline(Opts, SrcMgr, CB);
 
   // Create the hardware units defining the backend.
   auto RCU = std::make_unique<RetireControlUnit>(SM);
@@ -69,13 +70,14 @@ Context::createDefaultPipeline(const PipelineOptions &Opts, SourceMgr &SrcMgr) {
 }
 
 std::unique_ptr<Pipeline>
-Context::createInOrderPipeline(const PipelineOptions &Opts, SourceMgr &SrcMgr) {
+Context::createInOrderPipeline(const PipelineOptions &Opts, SourceMgr &SrcMgr,
+                               CustomBehaviour &CB) {
   const MCSchedModel &SM = STI.getSchedModel();
   auto PRF = std::make_unique<RegisterFile>(SM, MRI, Opts.RegisterFileSize);
 
   // Create the pipeline stages.
   auto Entry = std::make_unique<EntryStage>(SrcMgr);
-  auto InOrderIssue = std::make_unique<InOrderIssueStage>(STI, *PRF);
+  auto InOrderIssue = std::make_unique<InOrderIssueStage>(STI, *PRF, CB);
   auto StagePipeline = std::make_unique<Pipeline>();
 
   // Pass the ownership of all the hardware units to this Context.
diff --git a/llvm/lib/MCA/CustomBehaviour.cpp b/llvm/lib/MCA/CustomBehaviour.cpp
new file mode 100644
index 0000000..b160f4c
--- /dev/null
+++ b/llvm/lib/MCA/CustomBehaviour.cpp
@@ -0,0 +1,26 @@
+//===--------------------- CustomBehaviour.cpp ------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+/// \file
+///
+/// This file implements methods from the CustomBehaviour interface.
+///
+//===----------------------------------------------------------------------===//
+
+#include "llvm/MCA/CustomBehaviour.h"
+
+namespace llvm {
+namespace mca {
+
+unsigned CustomBehaviour::checkCustomHazard(ArrayRef<InstRef> IssuedInst,
+                                            const InstRef &IR) {
+  // 0 signifies that there are no hazards that need to be waited on
+  return 0;
+}
+
+} // namespace mca
+} // namespace llvm
diff --git a/llvm/lib/MCA/InstrBuilder.cpp b/llvm/lib/MCA/InstrBuilder.cpp
index fa11beb..50ca5e9 100644
--- a/llvm/lib/MCA/InstrBuilder.cpp
+++ b/llvm/lib/MCA/InstrBuilder.cpp
@@ -603,7 +603,8 @@ InstrBuilder::createInstruction(const MCInst &MCI) {
   if (!DescOrErr)
     return DescOrErr.takeError();
   const InstrDesc &D = *DescOrErr;
-  std::unique_ptr<Instruction> NewIS = std::make_unique<Instruction>(D);
+  std::unique_ptr<Instruction> NewIS =
+      std::make_unique<Instruction>(D, MCI.getOpcode());
 
   // Check if this is a dependency breaking instruction.
   APInt Mask;
diff --git a/llvm/lib/MCA/Stages/InOrderIssueStage.cpp b/llvm/lib/MCA/Stages/InOrderIssueStage.cpp
index b24ddbee4..a5dad7c 100644
--- a/llvm/lib/MCA/Stages/InOrderIssueStage.cpp
+++ b/llvm/lib/MCA/Stages/InOrderIssueStage.cpp
@@ -43,8 +43,8 @@ void StallInfo::cycleEnd() {
 }
 
 InOrderIssueStage::InOrderIssueStage(const MCSubtargetInfo &STI,
-                                     RegisterFile &PRF)
-    : STI(STI), PRF(PRF), RM(STI.getSchedModel()), NumIssued(), SI(),
+                                     RegisterFile &PRF, CustomBehaviour &CB)
+    : STI(STI), PRF(PRF), RM(STI.getSchedModel()), CB(CB), NumIssued(), SI(),
       CarryOver(), Bandwidth(), LastWriteBackCycle() {}
 
 unsigned InOrderIssueStage::getIssueWidth() const {
@@ -125,6 +125,11 @@ bool InOrderIssueStage::canExecute(const InstRef &IR) {
     return false;
   }
 
+  if (unsigned CustomStallCycles = CB.checkCustomHazard(IssuedInst, IR)) {
+    SI.update(IR, CustomStallCycles, StallInfo::StallKind::CUSTOM_STALL);
+    return false;
+  }
+
   if (LastWriteBackCycle) {
     if (!IR.getInstruction()->getDesc().RetireOOO) {
       unsigned NextWriteBackCycle = findFirstWriteBackCycle(IR);
@@ -333,6 +338,11 @@ void InOrderIssueStage::notifyStallEvent() {
         HWPressureEvent(HWPressureEvent::RESOURCES, IR));
     break;
   }
+  case StallInfo::StallKind::CUSTOM_STALL: {
+    notifyEvent<HWStallEvent>(
+        HWStallEvent(HWStallEvent::CustomBehaviourStall, IR));
+    break;
+  }
   }
 }
 
diff --git a/llvm/tools/llvm-mca/CMakeLists.txt b/llvm/tools/llvm-mca/CMakeLists.txt
index 9df1923..1661ab9 100644
--- a/llvm/tools/llvm-mca/CMakeLists.txt
+++ b/llvm/tools/llvm-mca/CMakeLists.txt
@@ -1,5 +1,7 @@
 include_directories(include)
 
+add_subdirectory(lib)
+
 set(LLVM_LINK_COMPONENTS
   AllTargetsAsmParsers
   AllTargetsDescs
@@ -30,3 +32,7 @@ add_llvm_tool(llvm-mca
   )
 
 set(LLVM_MCA_SOURCE_DIR ${CURRENT_SOURCE_DIR})
+
+target_link_libraries(llvm-mca PRIVATE
+  ${LLVM_MCA_CUSTOMBEHAVIOUR_TARGETS}
+  )
diff --git a/llvm/tools/llvm-mca/Views/DispatchStatistics.cpp b/llvm/tools/llvm-mca/Views/DispatchStatistics.cpp
index a1c0cf2..d5e4171 100644
--- a/llvm/tools/llvm-mca/Views/DispatchStatistics.cpp
+++ b/llvm/tools/llvm-mca/Views/DispatchStatistics.cpp
@@ -77,6 +77,8 @@ void DispatchStatistics::printDispatchStalls(raw_ostream &OS) const {
   printStalls(SS, HWStalls[HWStallEvent::StoreQueueFull], NumCycles);
   SS << "\nGROUP   - Static restrictions on the dispatch group: ";
   printStalls(SS, HWStalls[HWStallEvent::DispatchGroupStall], NumCycles);
+  SS << "\nUSH     - Uncategorised Structural Hazard:           ";
+  printStalls(SS, HWStalls[HWStallEvent::CustomBehaviourStall], NumCycles);
   SS << '\n';
   SS.flush();
   OS << Buffer;
diff --git a/llvm/tools/llvm-mca/lib/AMDGPU/AMDGPUCustomBehaviour.cpp b/llvm/tools/llvm-mca/lib/AMDGPU/AMDGPUCustomBehaviour.cpp
new file mode 100644
index 0000000..a655f3f
--- /dev/null
+++ b/llvm/tools/llvm-mca/lib/AMDGPU/AMDGPUCustomBehaviour.cpp
@@ -0,0 +1,33 @@
+//===------------------ AMDGPUCustomBehaviour.cpp ---------------*-C++ -* -===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+/// \file
+///
+/// This file implements methods from the AMDGPUCustomBehaviour class.
+///
+//===----------------------------------------------------------------------===//
+
+#include "AMDGPUCustomBehaviour.h"
+#include "MCTargetDesc/AMDGPUMCTargetDesc.h"
+#include "SIInstrInfo.h"
+#include "llvm/Support/WithColor.h"
+
+namespace llvm {
+namespace mca {
+
+AMDGPUCustomBehaviour::AMDGPUCustomBehaviour(const MCSubtargetInfo &STI,
+                                             const SourceMgr &SrcMgr,
+                                             const MCInstrInfo &MCII)
+    : CustomBehaviour(STI, SrcMgr, MCII) {}
+
+unsigned AMDGPUCustomBehaviour::checkCustomHazard(ArrayRef<InstRef> IssuedInst,
+                                                  const InstRef &IR) {
+  return 0;
+}
+
+} // namespace mca
+} // namespace llvm
diff --git a/llvm/tools/llvm-mca/lib/AMDGPU/AMDGPUCustomBehaviour.h b/llvm/tools/llvm-mca/lib/AMDGPU/AMDGPUCustomBehaviour.h
new file mode 100644
index 0000000..0dd21c7
--- /dev/null
+++ b/llvm/tools/llvm-mca/lib/AMDGPU/AMDGPUCustomBehaviour.h
@@ -0,0 +1,57 @@
+//===------------------- AMDGPUCustomBehaviour.h ----------------*-C++ -* -===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+/// \file
+///
+/// This file defines the AMDGPUCustomBehaviour class which inherits from
+/// CustomBehaviour.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TOOLS_LLVM_MCA_LIB_AMDGPU_AMDGPUCUSTOMBEHAVIOUR_H
+#define LLVM_TOOLS_LLVM_MCA_LIB_AMDGPU_AMDGPUCUSTOMBEHAVIOUR_H
+
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/MCA/CustomBehaviour.h"
+#include "llvm/Support/TargetParser.h"
+
+namespace llvm {
+namespace mca {
+
+class AMDGPUInstrPostProcess : public InstrPostProcess {
+public:
+  AMDGPUInstrPostProcess(const MCSubtargetInfo &STI, const MCInstrInfo &MCII)
+      : InstrPostProcess(STI, MCII) {}
+
+  ~AMDGPUInstrPostProcess() {}
+
+  void postProcessInstruction(std::unique_ptr<Instruction> &Inst,
+                              const MCInst &MCI) override {}
+};
+
+class AMDGPUCustomBehaviour : public CustomBehaviour {
+public:
+  AMDGPUCustomBehaviour(const MCSubtargetInfo &STI, const SourceMgr &SrcMgr,
+                        const MCInstrInfo &MCII);
+
+  ~AMDGPUCustomBehaviour() {}
+
+  /// This method is used to determine if an instruction
+  /// should be allowed to be dispatched. The return value is
+  /// how many cycles until the instruction can be dispatched.
+  /// This method is called after MCA has already checked for
+  /// register and hardware dependencies so this method should only
+  /// implement custom behaviour and dependencies that are not picked up
+  /// by MCA naturally.
+  unsigned checkCustomHazard(ArrayRef<InstRef> IssuedInst,
+                             const InstRef &IR) override;
+};
+
+} // namespace mca
+} // namespace llvm
+
+#endif /* LLVM_TOOLS_LLVM_MCA_LIB_AMDGPU_AMDGPUCUSTOMBEHAVIOUR_H */
diff --git a/llvm/tools/llvm-mca/lib/AMDGPU/CMakeLists.txt b/llvm/tools/llvm-mca/lib/AMDGPU/CMakeLists.txt
new file mode 100644
index 0000000..19c3a8f
--- /dev/null
+++ b/llvm/tools/llvm-mca/lib/AMDGPU/CMakeLists.txt
@@ -0,0 +1,17 @@
+include_directories(
+  ${LLVM_MAIN_SRC_DIR}/lib/Target/AMDGPU
+  ${LLVM_BINARY_DIR}/lib/Target/AMDGPU
+  )
+
+set(LLVM_LINK_COMPONENTS
+  AMDGPU
+  Core
+  Support
+  )
+
+add_llvm_library(LLVMMCACustomBehaviourAMDGPU
+  AMDGPUCustomBehaviour.cpp
+
+  DEPENDS
+  AMDGPUCommonTableGen
+  )
diff --git a/llvm/tools/llvm-mca/lib/CMakeLists.txt b/llvm/tools/llvm-mca/lib/CMakeLists.txt
new file mode 100644
index 0000000..75f639e
--- /dev/null
+++ b/llvm/tools/llvm-mca/lib/CMakeLists.txt
@@ -0,0 +1,8 @@
+set(TARGETS_TO_APPEND "")
+
+if (LLVM_TARGETS_TO_BUILD MATCHES "AMDGPU")
+  add_subdirectory(AMDGPU)
+  list(APPEND TARGETS_TO_APPEND LLVMMCACustomBehaviourAMDGPU)
+endif()
+
+set(LLVM_MCA_CUSTOMBEHAVIOUR_TARGETS ${TARGETS_TO_APPEND} PARENT_SCOPE)
diff --git a/llvm/tools/llvm-mca/llvm-mca.cpp b/llvm/tools/llvm-mca/llvm-mca.cpp
index 3bb2f10..df19c74 100644
--- a/llvm/tools/llvm-mca/llvm-mca.cpp
+++ b/llvm/tools/llvm-mca/llvm-mca.cpp
@@ -32,6 +32,7 @@
 #include "Views/SchedulerStatistics.h"
 #include "Views/SummaryView.h"
 #include "Views/TimelineView.h"
+#include "lib/AMDGPU/AMDGPUCustomBehaviour.h"
 #include "llvm/MC/MCAsmBackend.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/MCCodeEmitter.h"
@@ -42,6 +43,7 @@
 #include "llvm/MC/MCTargetOptionsCommandFlags.h"
 #include "llvm/MCA/CodeEmitter.h"
 #include "llvm/MCA/Context.h"
+#include "llvm/MCA/CustomBehaviour.h"
 #include "llvm/MCA/InstrBuilder.h"
 #include "llvm/MCA/Pipeline.h"
 #include "llvm/MCA/Stages/EntryStage.h"
@@ -220,6 +222,12 @@ static cl::opt<bool> ShowEncoding(
     cl::desc("Print encoding information in the instruction info view"),
     cl::cat(ViewOptions), cl::init(false));
 
+static cl::opt<bool> DisableCustomBehaviour(
+    "disable-cb",
+    cl::desc(
+        "Disable custom behaviour (use the default class which does nothing)."),
+    cl::cat(ViewOptions), cl::init(false));
+
 namespace {
 
 const Target *getTarget(const char *ProgName) {
@@ -285,6 +293,39 @@ static void processViewOptions(bool IsOutOfOrder) {
     processOptionImpl(PrintRetireStats, Default);
 }
 
+std::unique_ptr<mca::InstrPostProcess>
+createInstrPostProcess(const Triple &TheTriple, const MCSubtargetInfo &STI,
+                       const MCInstrInfo &MCII) {
+  // Might be a good idea to have a separate flag so that InstrPostProcess
+  // can be used with or without CustomBehaviour
+  if (DisableCustomBehaviour)
+    return std::make_unique<mca::InstrPostProcess>(STI, MCII);
+
+  if (TheTriple.isAMDGPU())
+    return std::make_unique<mca::AMDGPUInstrPostProcess>(STI, MCII);
+
+  return std::make_unique<mca::InstrPostProcess>(STI, MCII);
+}
+
+std::unique_ptr<mca::CustomBehaviour>
+createCustomBehaviour(const Triple &TheTriple, const MCSubtargetInfo &STI,
+                      const mca::SourceMgr &SrcMgr, const MCInstrInfo &MCII) {
+  // Build the appropriate CustomBehaviour object for the current target.
+  // The CustomBehaviour class should never depend on the source code,
+  // but it can depend on the list of mca::Instruction and any classes
+  // that can be built using just the target info. If you need extra
+  // information from the source code or the list of MCInst, consider
+  // adding that information to the mca::Instruction class and setting
+  // it during InstrBuilder::createInstruction().
+  if (DisableCustomBehaviour)
+    return std::make_unique<mca::CustomBehaviour>(STI, SrcMgr, MCII);
+
+  if (TheTriple.isAMDGPU())
+    return std::make_unique<mca::AMDGPUCustomBehaviour>(STI, SrcMgr, MCII);
+
+  return std::make_unique<mca::CustomBehaviour>(STI, SrcMgr, MCII);
+}
+
 // Returns true on success.
 static bool runPipeline(mca::Pipeline &P) {
   // Handle pipeline errors here.
@@ -498,6 +539,8 @@ int main(int argc, char **argv) {
     // Lower the MCInst sequence into an mca::Instruction sequence.
     ArrayRef<MCInst> Insts = Region->getInstructions();
     mca::CodeEmitter CE(*STI, *MAB, *MCE, Insts);
+    std::unique_ptr<mca::InstrPostProcess> IPP =
+        createInstrPostProcess(TheTriple, *STI, *MCII);
     std::vector<std::unique_ptr<mca::Instruction>> LoweredSequence;
     for (const MCInst &MCI : Insts) {
       Expected<std::unique_ptr<mca::Instruction>> Inst =
@@ -520,6 +563,8 @@ int main(int argc, char **argv) {
         return 1;
       }
 
+      IPP->postProcessInstruction(Inst.get(), MCI);
+
       LoweredSequence.emplace_back(std::move(Inst.get()));
     }
 
@@ -547,8 +592,17 @@ int main(int argc, char **argv) {
       continue;
     }
 
+    // Create the CustomBehaviour object for enforcing Target Specific
+    // behaviours and dependencies that aren't expressed well enough
+    // in the tablegen. CB cannot depend on the list of MCInst or
+    // the source code (but it can depend on the list of
+    // mca::Instruction or any objects that can be reconstructed
+    // from the target information).
+    std::unique_ptr<mca::CustomBehaviour> CB =
+        createCustomBehaviour(TheTriple, *STI, S, *MCII);
+
     // Create a basic pipeline simulating an out-of-order backend.
-    auto P = MCA.createDefaultPipeline(PO, S);
+    auto P = MCA.createDefaultPipeline(PO, S, *CB);
     mca::PipelinePrinter Printer(*P, PrintJson ? mca::View::OK_JSON
                                                : mca::View::OK_READABLE);