🦀
Rustコードにコメントを付与するGPT-3.5 TurboのFine-tuningモデルの比較
コードにコメントをちゃんと書くのが苦手でここ半年ほどはChatGPT、GitHub Copilot、DeepLを駆使してコメントをでっちあげています。
以前よりもコメントを楽に書けてはいるのですが、もう少し楽にコメントを書きたいという思いがあります。
特にRustではGuidelineがあり、これに則ったコメントはChatGPTで書くことができません。
もちろんプロンプトに凝ればある程度の質のコメントを生成できますが、コードに合わせてプロンプトを調整するのかかなりの手間です。
そこでRustのスタンダードライブラリのコードから下記のようなデータセットを作成し、それを元にGPT-3.5 TurboのFine-tuningモデルを作成してみます。
{
"messages": [
{
"role": "user",
"content": "{コードからコメントを除いたコード}"
},
{
"role": "assitant",
"content": "{オリジナルのコード}"
}
]
}
このデータセットの生成は使い捨てのコードなので記載はしませんが、ざっくりコードの流れを記載すると
- walkdir crateで対象のディレクトリのrsファイルを舐める
-
syn、quoteでファイルをAST化後に文字列に変換
a. ASTレベルでdocコメントを削除しようとしたが上手くいかなかった - 正規表現で
#\s*\[\s*doc\s*=\s*".*"\s*]に一致するのを削除
というようなコードでサクッと作っています。
コメントの生成方法
Fine-tuningモデルをすぐに使えるということでプレイグラウンドでコメントの生成を行います。
system
Add a doc comment to the Rust code entered by the user.
user
use crate::alloc::{AllocError, Allocator};
use crate::ptr::RawPtr;
use std::alloc::{Layout, System};
use std::ops::{Deref, DerefMut};
use std::ptr::NonNull;
pub struct OwnedPtr<T, A = System>
where
T: Sized,
A: Allocator,
{
pub(in crate::ptr) ptr: NonNull<T>,
pub(in crate::ptr) size: usize,
alloc: A,
}
impl<T, A> OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
pub unsafe fn as_slice(&self) -> &[T] {
assert_eq!(
self.size % std::mem::size_of::<T>(),
0,
"Invalid size field"
);
let ptr = self.ptr.as_ptr() as *const T;
let len = self.size / std::mem::size_of::<T>();
std::slice::from_raw_parts(ptr, len)
}
pub unsafe fn as_slice_mut(&mut self) -> &mut [T] {
assert_eq!(
self.size % std::mem::size_of::<T>(),
0,
"Invalid size field"
);
let ptr = self.ptr.as_ptr();
let len = self.size / std::mem::size_of::<T>();
std::slice::from_raw_parts_mut(ptr, len)
}
pub fn into_raw(self) -> RawPtr<T> {
let ptr = self.ptr;
let size = self.size;
std::mem::forget(self);
RawPtr { ptr, size }
}
pub fn into_raw_slice(self) -> RawPtr<[T]> {
let ptr = self.ptr.as_ptr();
let size = self.size;
std::mem::forget(self);
unsafe { RawPtr::from_raw_parts(ptr, size) }
}
}
impl<T> OwnedPtr<T, System>
where
T: Sized,
{
pub fn new(x: T) -> Self {
let boxed = Box::new(x);
let ptr = Box::into_raw(boxed);
Self {
ptr: NonNull::new(ptr).expect("can never be NULL as it is a pointer to a valid T"),
size: std::mem::size_of::<T>(),
alloc: System,
}
}
pub fn from_vec(mut x: Vec<T>) -> Self {
let ptr = x.as_mut_ptr();
let size = x.len() * std::mem::size_of::<T>();
std::mem::forget(x);
Self {
ptr: NonNull::new(ptr).expect("can never be NULL as it is a pointer to a valid Vec<T>"),
size,
alloc: System,
}
}
pub fn from_slice(x: &[T]) -> Self
where
T: Clone,
{
let x = x.to_vec();
Self::from_vec(x)
}
}
impl OwnedPtr<u8, System> {
pub fn reallocate<R: Allocator + Default>(self) -> Result<OwnedPtr<u8, R>, AllocError> {
let alloc = R::default();
let layout = Layout::from_size_align(self.size, std::mem::align_of::<u8>())
.expect("`Layout` can always be created with align_of created align");
let ptr = alloc.allocate(layout)?;
let src = self.ptr.as_ptr() as *const u8;
let dest = ptr.as_ptr() as *mut u8;
unsafe {
std::ptr::copy_nonoverlapping(src, dest, self.size);
};
Ok(OwnedPtr {
ptr: ptr.cast(),
size: self.size,
alloc,
})
}
}
impl<T, A> Deref for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
type Target = T;
fn deref(&self) -> &Self::Target {
self.as_ref()
}
}
impl<T, A> DerefMut for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn deref_mut(&mut self) -> &mut Self::Target {
self.as_mut()
}
}
impl<T, A> AsRef<T> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn as_ref(&self) -> &T {
unsafe { self.ptr.as_ref() }
}
}
impl<T, A> AsMut<T> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn as_mut(&mut self) -> &mut T {
unsafe { self.ptr.as_mut() }
}
}
impl<T, A> Drop for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn drop(&mut self) {
let layout = Layout::from_size_align(self.size, std::mem::align_of::<T>()).unwrap();
self.alloc.deallocate(self.ptr.cast(), layout);
}
}
impl<T, A> From<RawPtr<T>> for OwnedPtr<T, A>
where
A: Allocator + Default,
{
fn from(value: RawPtr<T>) -> Self {
Self {
ptr: value.ptr,
size: value.size,
alloc: A::default(),
}
}
}
impl<T, A> From<RawPtr<[T]>> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator + Default,
{
fn from(value: RawPtr<[T]>) -> Self {
Self {
ptr: value.ptr.cast(),
size: value.size,
alloc: A::default(),
}
}
}
impl<T, A> From<OwnedPtr<T, A>> for RawPtr<T>
where
A: Allocator + Default,
{
fn from(value: OwnedPtr<T, A>) -> Self {
value.into_raw()
}
}
impl<T, A> From<OwnedPtr<T, A>> for RawPtr<[T]>
where
A: Allocator + Default,
{
fn from(value: OwnedPtr<T, A>) -> Self {
value.into_raw_slice()
}
}
- Temperature:
0.45 - Maximum length:
2048 - Top P:
1 - Frequency penalty:
0 - Presence penalty:
0
上記の設定値でモデルを変えて生成していきます。
コメントをつけるコードのオリジナルは上記で、ちょうどExamples、Safetyセクションの記載ができそうなのでちょうど良さそうという考えで採用しています。
gpt-3.5-turbo
/// A wrapper around a raw pointer that owns the memory it points to.
///
/// This struct provides safe access to the memory through methods like `as_slice` and `as_slice_mut`.
/// It also takes care of deallocating the memory when it is dropped.
///
/// # Type Parameters
///
/// - `T`: The type of the data that the pointer points to.
/// - `A`: The allocator used to allocate and deallocate the memory. Defaults to `std::alloc::System`.
///
/// # Fields
///
/// - `ptr`: The non-null raw pointer to the memory.
/// - `size`: The size of the allocated memory in bytes.
/// - `alloc`: The allocator used to allocate and deallocate the memory.
///
/// # Safety
///
/// The `OwnedPtr` struct assumes ownership of the memory pointed to by the raw pointer.
/// It is responsible for deallocating the memory when it is dropped.
/// The `as_slice` and `as_slice_mut` methods assume that the size field is valid and correctly represents
/// the size of the allocated memory in bytes.
pub struct OwnedPtr<T, A = System>
where
T: Sized,
A: Allocator,
{
pub(in crate::ptr) ptr: NonNull<T>,
pub(in crate::ptr) size: usize,
alloc: A,
}
impl<T, A> OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
/// Returns a shared reference to the slice of `T` values.
///
/// # Safety
///
/// This method assumes that the size field is valid and correctly represents the size of the allocated memory in bytes.
/// It also assumes that the memory is properly aligned and initialized.
pub unsafe fn as_slice(&self) -> &[T] {
assert_eq!(
self.size % std::mem::size_of::<T>(),
0,
"Invalid size field"
);
let ptr = self.ptr.as_ptr() as *const T;
let len = self.size / std::mem::size_of::<T>();
std::slice::from_raw_parts(ptr, len)
}
/// Returns a mutable reference to the slice of `T` values.
///
/// # Safety
///
/// This method assumes that the size field is valid and correctly represents the size of the allocated memory in bytes.
/// It also assumes that the memory is properly aligned and initialized.
pub unsafe fn as_slice_mut(&mut self) -> &mut [T] {
assert_eq!(
self.size % std::mem::size_of::<T>(),
0,
"Invalid size field"
);
let ptr = self.ptr.as_ptr();
let len = self.size / std::mem::size_of::<T>();
std::slice::from_raw_parts_mut(ptr, len)
}
/// Converts the `OwnedPtr` into a `RawPtr<T>`.
///
/// This method transfers ownership of the memory to the resulting `RawPtr`.
/// The `OwnedPtr` is forgotten and will not deallocate the memory when dropped.
pub fn into_raw(self) -> RawPtr<T> {
let ptr = self.ptr;
let size = self.size;
std::mem::forget(self);
RawPtr { ptr, size }
}
/// Converts the `OwnedPtr` into a `RawPtr<[T]>`.
///
/// This method transfers ownership of the memory to the resulting `RawPtr`.
/// The `OwnedPtr` is forgotten and will not deallocate the memory when dropped.
pub fn into_raw_slice(self) -> RawPtr<[T]> {
let ptr = self.ptr.as_ptr();
let size = self.size;
std::mem::forget(self);
unsafe { RawPtr::from_raw_parts(ptr, size) }
}
}
impl<T> OwnedPtr<T, System>
where
T: Sized,
{
/// Creates a new `OwnedPtr` from a value of type `T`.
///
/// The value is moved into a `Box` and the resulting raw pointer is stored in the `OwnedPtr`.
pub fn new(x: T) -> Self {
let boxed = Box::new(x);
let ptr = Box::into_raw(boxed);
Self {
ptr: NonNull::new(ptr).expect("can never be NULL as it is a pointer to a valid T"),
size: std::mem::size_of::<T>(),
alloc: System,
}
}
/// Creates a new `OwnedPtr` from a `Vec<T>`.
///
/// The `Vec` is converted into a raw pointer and the resulting raw pointer is stored in the `OwnedPtr`.
/// The `Vec` is forgotten and will not deallocate the memory when dropped.
pub fn from_vec(mut x: Vec<T>) -> Self {
let ptr = x.as_mut_ptr();
let size = x.len() * std::mem::size_of::<T>();
std::mem::forget(x);
Self {
ptr: NonNull::new(ptr).expect("can never be NULL as it is a pointer to a valid Vec<T>"),
size,
alloc: System,
}
}
/// Creates a new `OwnedPtr` from a slice of `T` values.
///
/// The slice is converted into a `Vec` and then into a raw pointer.
/// The resulting raw pointer is stored in the `OwnedPtr`.
/// The `Vec` is forgotten and will not deallocate the memory when dropped.
pub fn from_slice(x: &[T]) -> Self
where
T: Clone,
{
let x = x.to_vec();
Self::from_vec(x)
}
}
impl OwnedPtr<u8, System> {
/// Reallocates the memory using a different allocator.
///
/// This method creates a new `OwnedPtr` with the same size as the original,
/// but using a different allocator specified by the type parameter `R`.
/// The memory is copied from the original memory to the new memory.
///
/// # Errors
///
/// This method returns an `AllocError` if the allocation fails.
pub fn reallocate<R: Allocator + Default>(self) -> Result<OwnedPtr<u8, R>, AllocError> {
let alloc = R::default();
let layout = Layout::from_size_align(self.size, std::mem::align_of::<u8>())
.expect("`Layout` can always be created with align_of created align");
let ptr = alloc.allocate(layout)?;
let src = self.ptr.as_ptr() as *const u8;
let dest = ptr.as_ptr() as *mut u8;
unsafe {
std::ptr::copy_nonoverlapping(src, dest, self.size);
};
Ok(OwnedPtr {
ptr: ptr.cast(),
size: self.size,
alloc,
})
}
}
impl<T, A> Deref for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
type Target = T;
fn deref(&self) -> &Self::Target {
self.as_ref()
}
}
impl<T, A> DerefMut for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn deref_mut(&mut self) -> &mut Self::Target {
self.as_mut()
}
}
impl<T, A> AsRef<T> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn as_ref(&self) -> &T {
unsafe { self.ptr.as_ref() }
}
}
impl<T, A> AsMut<T> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn as_mut(&mut self) -> &mut T {
unsafe { self.ptr.as_mut() }
}
}
impl<T, A> Drop for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn drop(&mut self) {
let layout = Layout::from_size_align(self.size, std::mem::align_of::<T>()).unwrap();
self.alloc.deallocate(self.ptr.cast(), layout);
}
}
impl<T, A> From<RawPtr<T>> for OwnedPtr<T, A>
where
A: Allocator + Default,
{
fn from(value: RawPtr<T>) -> Self {
Self {
ptr: value.ptr,
size: value.size,
alloc: A::default(),
}
}
}
impl<T, A> From<RawPtr<[T]>> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator + Default,
{
fn from(value: RawPtr<[T]>) -> Self {
Self {
ptr: value.ptr.cast(),
size: value.size,
alloc: A::default(),
}
}
}
impl<T, A> From<OwnedPtr<T, A>> for RawPtr<T>
where
A: Allocator + Default,
{
fn from(value: OwnedPtr<T, A>) -> Self {
value.into_raw()
}
}
impl<T, A> From<OwnedPtr<T, A>> for RawPtr<[T]>
where
A: Allocator + Default,
{
fn from(value: OwnedPtr<T, A>) -> Self {
value.into_raw_slice()
}
}
gpt-3.5-turbo-16k
/// A wrapper around a non-null pointer to owned memory, with support for deallocation.
///
/// This struct is generic over the type `T` of the owned memory and the allocator `A` used to allocate and deallocate the memory. The `T` type must be `Sized`, and the `A` type must implement the `Allocator` trait.
///
/// # Examples
///
/// ```
/// use std::alloc::System;
/// use std::ptr::NonNull;
///
/// let ptr = NonNull::dangling();
/// let size = 0;
/// let alloc = System;
///
/// let owned_ptr = OwnedPtr { ptr, size, alloc };
/// ```
pub struct OwnedPtr<T, A = System>
where
T: Sized,
A: Allocator,
{
pub(in crate::ptr) ptr: NonNull<T>,
pub(in crate::ptr) size: usize,
alloc: A,
}
impl<T, A> OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
/// Returns a shared reference to the owned memory as a slice.
///
/// # Safety
///
/// This function is unsafe because it performs no bounds checking on the memory slice. The caller must ensure that the slice does not outlive the owned memory.
pub unsafe fn as_slice(&self) -> &[T] {
assert_eq!(
self.size % std::mem::size_of::<T>(),
0,
"Invalid size field"
);
let ptr = self.ptr.as_ptr() as *const T;
let len = self.size / std::mem::size_of::<T>();
std::slice::from_raw_parts(ptr, len)
}
/// Returns a mutable reference to the owned memory as a slice.
///
/// # Safety
///
/// This function is unsafe because it performs no bounds checking on the memory slice. The caller must ensure that the slice does not outlive the owned memory, and that there are no other references to the owned memory.
pub unsafe fn as_slice_mut(&mut self) -> &mut [T] {
assert_eq!(
self.size % std::mem::size_of::<T>(),
0,
"Invalid size field"
);
let ptr = self.ptr.as_ptr();
let len = self.size / std::mem::size_of::<T>();
std::slice::from_raw_parts_mut(ptr, len)
}
/// Consumes the `OwnedPtr` and returns a `RawPtr` to the owned memory.
///
/// This function transfers ownership of the owned memory to the returned `RawPtr`. The `OwnedPtr` is forgotten and will not deallocate the memory when dropped.
pub fn into_raw(self) -> RawPtr<T> {
let ptr = self.ptr;
let size = self.size;
std::mem::forget(self);
RawPtr { ptr, size }
}
/// Consumes the `OwnedPtr` and returns a `RawPtr` to the owned memory as a slice.
///
/// This function transfers ownership of the owned memory to the returned `RawPtr`. The `OwnedPtr` is forgotten and will not deallocate the memory when dropped.
pub fn into_raw_slice(self) -> RawPtr<[T]> {
let ptr = self.ptr.as_ptr();
let size = self.size;
std::mem::forget(self);
unsafe { RawPtr::from_raw_parts(ptr, size) }
}
}
impl<T> OwnedPtr<T, System>
where
T: Sized,
{
/// Creates a new `OwnedPtr` from a value.
///
/// This function allocates memory for the value using the `System` allocator and returns an `OwnedPtr` wrapping the allocated memory.
pub fn new(x: T) -> Self {
let boxed = Box::new(x);
let ptr = Box::into_raw(boxed);
Self {
ptr: NonNull::new(ptr).expect("can never be NULL as it is a pointer to a valid T"),
size: std::mem::size_of::<T>(),
alloc: System,
}
}
/// Creates a new `OwnedPtr` from a `Vec`.
///
/// This function takes ownership of the `Vec` and returns an `OwnedPtr` wrapping the owned memory of the `Vec`.
pub fn from_vec(mut x: Vec<T>) -> Self {
let ptr = x.as_mut_ptr();
let size = x.len() * std::mem::size_of::<T>();
std::mem::forget(x);
Self {
ptr: NonNull::new(ptr).expect("can never be NULL as it is a pointer to a valid Vec<T>"),
size,
alloc: System,
}
}
/// Creates a new `OwnedPtr` from a slice.
///
/// This function clones the elements of the slice into a `Vec` and then calls `from_vec` to create the `OwnedPtr`.
pub fn from_slice(x: &[T]) -> Self
where
T: Clone,
{
let x = x.to_vec();
Self::from_vec(x)
}
}
impl OwnedPtr<u8, System> {
/// Reallocates the owned memory using a different allocator.
///
/// This function deallocates the current memory using the default allocator `R::default()`, and then allocates new memory using the same size and alignment as the current memory. The contents of the current memory are copied to the new memory, and the new memory is wrapped in a new `OwnedPtr` using the allocator `R`.
///
/// # Errors
///
/// This function returns an `AllocError` if the allocation fails.
pub fn reallocate<R: Allocator + Default>(self) -> Result<OwnedPtr<u8, R>, AllocError> {
let alloc = R::default();
let layout = Layout::from_size_align(self.size, std::mem::align_of::<u8>())
.expect("`Layout` can always be created with align_of created align");
let ptr = alloc.allocate(layout)?;
let src = self.ptr.as_ptr() as *const u8;
let dest = ptr.as_ptr() as *mut u8;
unsafe {
std::ptr::copy_nonoverlapping(src, dest, self.size);
};
Ok(OwnedPtr {
ptr: ptr.cast(),
size: self.size,
alloc,
})
}
}
impl<T, A> Deref for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
type Target = T;
fn deref(&self) -> &Self::Target {
self.as_ref()
}
}
impl<T, A> DerefMut for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn deref_mut(&mut self) -> &mut Self::Target {
self.as_mut()
}
}
impl<T, A> AsRef<T> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn as_ref(&self) -> &T {
unsafe { self.ptr.as_ref() }
}
}
impl<T, A> AsMut<T> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn as_mut(&mut self) -> &mut T {
unsafe { self.ptr.as_mut() }
}
}
impl<T, A> Drop for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn drop(&mut self) {
let layout = Layout::from_size_align(self.size, std::mem::align_of::<T>()).unwrap();
self.alloc.deallocate(self.ptr.cast(), layout);
}
}
impl<T, A> From<RawPtr<T>> for OwnedPtr<T, A>
where
A: Allocator + Default,
{
fn from(value: RawPtr<T>) -> Self {
Self {
ptr: value.ptr,
size: value.size,
alloc: A::default(),
}
}
}
impl<T, A> From<RawPtr<[T]>> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator + Default,
{
fn from(value: RawPtr<[T]>) -> Self {
Self {
ptr: value.ptr.cast(),
size: value.size,
alloc: A::default(),
}
}
}
impl<T, A> From<OwnedPtr<T, A>> for RawPtr<T>
where
A: Allocator + Default,
{
fn from(value: OwnedPtr<T, A>) -> Self {
value.into_raw()
}
}
impl<T, A> From<OwnedPtr<T, A>> for RawPtr<[T]>
where
A: Allocator + Default,
{
fn from(value: OwnedPtr<T, A>) -> Self {
value.into_raw_slice()
}
}
gpt-4
/// A custom smart pointer type `OwnedPtr` that owns the data it points to.
/// It uses an allocator `A` to allocate and deallocate memory.
/// The default allocator is `System`.
///
/// # Type Parameters
/// * `T`: The type of the data the pointer owns.
/// * `A`: The allocator type used to allocate and deallocate memory.
pub struct OwnedPtr<T, A = System>
where
T: Sized,
A: Allocator,
{
pub(in crate::ptr) ptr: NonNull<T>,
pub(in crate::ptr) size: usize,
alloc: A,
}
/// Implementation of `OwnedPtr`.
impl<T, A> OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
/// Returns a slice that points to the same memory location.
pub unsafe fn as_slice(&self) -> &[T] {...}
/// Returns a mutable slice that points to the same memory location.
pub unsafe fn as_slice_mut(&mut self) -> &mut [T] {...}
/// Consumes the `OwnedPtr`, returning the wrapped raw pointer.
pub fn into_raw(self) -> RawPtr<T> {...}
/// Consumes the `OwnedPtr`, returning a raw slice pointer.
pub fn into_raw_slice(self) -> RawPtr<[T]> {...}
}
/// Implementation of `OwnedPtr` for `System` allocator.
impl<T> OwnedPtr<T, System>
where
T: Sized,
{
/// Creates a new `OwnedPtr` that owns `x`.
pub fn new(x: T) -> Self {...}
/// Creates a new `OwnedPtr` from a vector.
pub fn from_vec(mut x: Vec<T>) -> Self {...}
/// Creates a new `OwnedPtr` from a slice.
pub fn from_slice(x: &[T]) -> Self
where
T: Clone,
{...}
}
/// Implementation of `OwnedPtr` for `u8` type and `System` allocator.
impl OwnedPtr<u8, System> {
/// Reallocates the memory block that the `OwnedPtr` points to.
pub fn reallocate<R: Allocator + Default>(self) -> Result<OwnedPtr<u8, R>, AllocError> {...}
}
/// Implementation of `Deref` trait for `OwnedPtr`.
impl<T, A> Deref for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
type Target = T;
fn deref(&self) -> &Self::Target {...}
}
/// Implementation of `DerefMut` trait for `OwnedPtr`.
impl<T, A> DerefMut for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn deref_mut(&mut self) -> &mut Self::Target {...}
}
/// Implementation of `AsRef` trait for `OwnedPtr`.
impl<T, A> AsRef<T> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn as_ref(&self) -> &T {...}
}
/// Implementation of `AsMut` trait for `OwnedPtr`.
impl<T, A> AsMut<T> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn as_mut(&mut self) -> &mut T {...}
}
/// Implementation of `Drop` trait for `OwnedPtr`.
impl<T, A> Drop for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn drop(&mut self) {...}
}
/// Implementation of `From` trait for `RawPtr<T>` to `OwnedPtr<T, A>`.
impl<T, A> From<RawPtr<T>> for OwnedPtr<T, A>
where
A: Allocator + Default,
{
fn from(value: RawPtr<T>) -> Self {...}
}
/// Implementation of `From` trait for `RawPtr<[T]>` to `OwnedPtr<T, A>`.
impl<T, A> From<RawPtr<[T]>> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator + Default,
{
fn from(value: RawPtr<[T]>) -> Self {...}
}
/// Implementation of `From` trait for `OwnedPtr<T, A>` to `RawPtr<T>`.
impl<T, A> From<OwnedPtr<T, A>> for RawPtr<T>
where
A: Allocator + Default,
{
fn from(value: OwnedPtr<T, A>) -> Self {...}
}
/// Implementation of `From` trait for `OwnedPtr<T, A>` to `RawPtr<[T]>`.
impl<T, A> From<OwnedPtr<T, A>> for RawPtr<[T]>
where
A: Allocator + Default,
{
fn from(value: OwnedPtr<T, A>) -> Self {...}
}
115パターン、428KBのデータセットでFine-tuningしたモデル
対象のコードが4,000文字以下を対象にしたデータモデル。
/// A pointer type for owned memory.
///
/// This type is a version of `Box` which is generic over the allocator. It is
/// only available when the `alloc` feature of this library is activated, and
/// it is identical to `Box` otherwise.
///
/// See the [owned pointer documentation][owned] for details.
///
/// [owned]: ../../std/boxed/struct.Box.html
use crate::alloc::{AllocError, Allocator};
use crate::ptr::RawPtr;
use std::alloc::{Layout, System};
use std::ops::{Deref, DerefMut};
use std::ptr::NonNull;
/// A pointer type for owned memory.
///
/// This type is a version of `Box` which is generic over the allocator. It is
/// only available when the `alloc` feature of this library is activated, and
/// it is identical to `Box` otherwise.
///
/// See the [owned pointer documentation][owned] for details.
///
/// [owned]: ../../std/boxed/struct.Box.html
pub struct OwnedPtr<T, A = System>
where
T: Sized,
A: Allocator,
{
pub(in crate::ptr) ptr: NonNull<T>,
pub(in crate::ptr) size: usize,
alloc: A,
}
impl<T, A> OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
/// Constructs a new instance of `Box<T, A>` from a pointer and a size.
///
/// This function will take ownership of the provided `ptr`, and it will be
/// deallocated when the returned `Box` is deallocated. Additionally the
/// returned box will have the provided layout.
///
/// # Safety
///
/// This function is unsafe as improper use may lead to memory problems. For
/// example a double-free may occur if the function is called twice on the
/// same raw pointer.
pub unsafe fn from_raw_parts(ptr: *mut T, size: usize) -> Self {
OwnedPtr {
ptr: NonNull::new(ptr).expect("can never be NULL as it is a pointer to a valid T"),
size,
alloc: A::default(),
}
}
pub unsafe fn as_slice(&self) -> &[T] {
assert_eq!(
self.size % std::mem::size_of::<T>(),
0,
"Invalid size field"
);
let ptr = self.ptr.as_ptr() as *const T;
let len = self.size / std::mem::size_of::<T>();
std::slice::from_raw_parts(ptr, len)
}
pub unsafe fn as_slice_mut(&mut self) -> &mut [T] {
assert_eq!(
self.size % std::mem::size_of::<T>(),
0,
"Invalid size field"
);
let ptr = self.ptr.as_ptr();
let len = self.size / std::mem::size_of::<T>();
std::slice::from_raw_parts_mut(ptr, len)
}
/// Consumes the `Box`, returning the wrapped raw pointer.
///
/// After calling this function, the caller is responsible for the memory
/// previously managed by the `Box`. Additionally, the `Box` itself is no
/// longer subject to RAII.
///
/// The caller must ensure that the memory is properly deallocated. Specifically, for
/// `Box` allocated with the standard library, this means that [`dealloc`] must be
/// called with the appropriate [memory layout] and allocator used.
///
/// # Example
///
/// ```rust
/// use std::boxed::Box;
///
/// let x = Box::new(5);
/// let ptr = Box::into_raw(x);
///
/// unsafe {
/// assert_eq!(*ptr, 5);
/// Box::from_raw(ptr); // retakes ownership, will destroy at end of scope
/// }
/// ```
///
/// [`dealloc`]: ../../std/alloc/trait.Alloc.html#tymethod.dealloc
/// [memory layout]: ../../std/alloc/struct.Layout.html
pub fn into_raw(self) -> RawPtr<T> {
let ptr = self.ptr;
let size = self.size;
std::mem::forget(self);
RawPtr { ptr, size }
}
/// Consumes the `Box`, returning a wrapped raw pointer.
///
/// Similar to [`into_raw`], but will additionally return the size of the
/// allocated buffer. This is useful when the `Box` wrapper is used to
/// allocate a buffer with the size unknown at compile-time.
///
/// After calling this function, the caller is responsible for the memory
/// previously managed by the `Box`. Additionally, the `Box` itself is no
/// longer subject to RAII.
///
/// The caller must ensure that the memory is properly deallocated. Specifically, for
/// `Box` allocated with the standard library, this means that [`dealloc`] must be
/// called with the appropriate [memory layout] and allocator used.
///
/// # Example
///
/// ```rust
/// use std::boxed::Box;
///
/// let x = vec![5, 6, 7];
/// let x = x.into_boxed_slice();
/// let (ptr, len) = Box::into_raw_with_len(x);
///
/// unsafe {
/// assert_eq!(vec![5, 6, 7], Vec::from_raw_parts(ptr, len, len));
/// }
/// ```
///
/// [`into_raw`]: ../../std/boxed/struct.Box.html#method.into_raw
/// [`dealloc`]: ../../std/alloc/trait.Alloc.html#tymethod.dealloc
/// [memory layout]: ../../std/alloc/struct.Layout.html
pub fn into_raw_with_len(self) -> RawPtr<[T]> {
let ptr = self.ptr.as_ptr();
let size = self.size;
std::mem::forget(self);
unsafe { RawPtr::from_raw_parts(ptr, size) }
}
}
impl<T> OwnedPtr<T, System>
where
T: Sized,
{
/// Constructs a new instance of `Box<T>` from an initial value of `T`.
///
/// This will allocate enough space for the `T` provided and then move the
/// provided value into the allocated space. The returned box will
/// represent ownership over the allocated `T` and responsibility for
/// deallocating it.
///
/// # Example
///
/// ```rust
/// use std::boxed::Box;
///
/// let five = Box::new(5);
///
/// assert_eq!(*five, 5);
/// ```
pub fn new(x: T) -> Self {
let boxed = Box::new(x);
let ptr = Box::into_raw(boxed);
Self {
ptr: NonNull::new(ptr).expect("can never be NULL as it is a pointer to a valid T"),
size: std::mem::size_of::<T>(),
alloc: System,
}
}
/// Constructs a new instance of `Box<T>` from a `Vec<T>`.
///
/// This will move the entire vector `x` into the allocated space, including
/// all of its elements. The returned box will represent ownership over
/// the allocated `T` and responsibility for deallocating it. Note that
/// this function consumes the vector passed to it.
///
/// # Example
///
/// ```rust
/// use std::boxed::Box;
///
/// let vec = vec![1, 2, 3, 4, 5];
/// let boxed = Box::from_vec(vec);
///
/// assert_eq!(&*boxed, &[1, 2, 3, 4, 5]);
/// ```
pub fn from_vec(mut x: Vec<T>) -> Self {
let ptr = x.as_mut_ptr();
let size = x.len() * std::mem::size_of::<T>();
std::mem::forget(x);
Self {
ptr: NonNull::new(ptr).expect("can never be NULL as it is a pointer to a valid Vec<T>"),
size,
alloc: System,
}
}
/// Constructs a new instance of `Box<T>` from a slice.
///
/// This will copy each element of the slice into a new instance of `T` and
/// then move all of these instances into a new instance of `Box`. The
/// resulting box will be the length of the slice and will have ownership
/// over all the copied elements of the slice.
///
/// # Example
///
/// ```rust
/// use std::boxed::Box;
///
/// let slice = &[1, 2, 3, 4, 5];
/// let boxed = Box::from_slice(slice);
///
/// assert_eq!(&*boxed, slice);
/// ```
pub fn from_slice(x: &[T]) -> Self
where
T: Clone,
{
let x = x.to_vec();
Self::from_vec(x)
}
}
impl OwnedPtr<u8, System> {
/// Reallocate memory.
///
/// # Safety
///
/// This function is unsafe because it changes the allocation of the memory
/// block. In particular, it is unsafe to call this function twice on the
241パターン、1.4MBのデータセットでFine-tuningしたモデル
対象のコードが8,000文字以下を対象にしたデータモデル。
/// A pointer type for use with custom allocators.
///
/// This type is analagous to [`Box<T>`], except that it allows the
/// user to specify an allocator other than the system allocator.
///
/// # Example
///
/// This example shows how to use a custom allocator with `OwnedPtr`.
///
/// ```rust
/// use std::alloc::{Alloc, Layout, System};
/// use owned_ptr::OwnedPtr;
///
/// struct MyAllocator;
///
/// unsafe impl Alloc for MyAllocator {
/// unsafe fn alloc(&self, layout: Layout) -> Result<*mut u8, std::alloc::AllocError> {
/// System.alloc(layout)
/// }
///
/// unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
/// System.dealloc(ptr, layout)
/// }
/// }
///
/// fn main() {
/// let allocator = MyAllocator;
/// let x = OwnedPtr::new_in(42, allocator);
/// assert_eq!(*x, 42);
/// }
/// ```
///
/// [`Box<T>`]: std::boxed::Box
use crate::alloc::{AllocError, Allocator};
use crate::ptr::RawPtr;
use std::alloc::{Layout, System};
use std::ops::{Deref, DerefMut};
use std::ptr::NonNull;
/// A pointer type for use with custom allocators.
///
/// This type is analagous to [`Box<T>`], except that it allows the
/// user to specify an allocator other than the system allocator.
///
/// # Example
///
/// This example shows how to use a custom allocator with `OwnedPtr`.
///
/// ```rust
/// use std::alloc::{Alloc, Layout, System};
/// use owned_ptr::OwnedPtr;
///
/// struct MyAllocator;
///
/// unsafe impl Alloc for MyAllocator {
/// unsafe fn alloc(&self, layout: Layout) -> Result<*mut u8, std::alloc::AllocError> {
/// System.alloc(layout)
/// }
///
/// unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
/// System.dealloc(ptr, layout)
/// }
/// }
///
/// fn main() {
/// let allocator = MyAllocator;
/// let x = OwnedPtr::new_in(42, allocator);
/// assert_eq!(*x, 42);
/// }
/// ```
///
/// [`Box<T>`]: std::boxed::Box
pub struct OwnedPtr<T, A = System>
where
T: Sized,
A: Allocator,
{
pub(in crate::ptr) ptr: NonNull<T>,
pub(in crate::ptr) size: usize,
alloc: A,
}
impl<T, A> OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
/// Unsafely create an `OwnedPtr` from a raw pointer and a size.
///
/// # Safety
///
/// The caller must ensure that the pointer is not null and that the
/// size is correct.
pub unsafe fn from_raw_parts(ptr: NonNull<T>, size: usize, alloc: A) -> Self {
Self { ptr, size, alloc }
}
/// Unsafely create an `OwnedPtr` from a raw pointer and a size.
///
/// # Safety
///
/// The caller must ensure that the pointer is not null and that the
/// size is correct.
pub unsafe fn from_raw_parts_in(ptr: NonNull<T>, size: usize, alloc: A) -> Self {
Self { ptr, size, alloc }
}
/// Unsafely create an `OwnedPtr` from a raw pointer and a size.
///
/// # Safety
///
/// The caller must ensure that the pointer is not null and that the
/// size is correct.
pub unsafe fn from_raw_parts_in_layout(
ptr: NonNull<T>,
size: usize,
layout: Layout,
alloc: A,
) -> Self {
assert_eq!(size, layout.size(), "size mismatch");
Self { ptr, size, alloc }
}
/// Unsafely create an `OwnedPtr` from a raw pointer and a size.
///
/// # Safety
///
/// The caller must ensure that the pointer is not null and that the
/// size is correct.
pub unsafe fn from_raw_parts_in_layout_zeroed(
ptr: NonNull<T>,
size: usize,
layout: Layout,
alloc: A,
) -> Self {
assert_eq!(size, layout.size(), "size mismatch");
let ptr = ptr.cast();
alloc.alloc_zeroed(layout).expect("allocation failed");
Self { ptr, size, alloc }
}
/// Unsafely create an `OwnedPtr` from a raw pointer and a size.
///
/// # Safety
///
/// The caller must ensure that the pointer is not null and that the
/// size is correct.
pub unsafe fn from_raw_parts_zeroed(ptr: NonNull<T>, size: usize, alloc: A) -> Self {
let layout = Layout::from_size_align_unchecked(size, std::mem::align_of::<T>());
alloc.alloc_zeroed(layout).expect("allocation failed");
Self { ptr, size, alloc }
}
/// Unsafely create an `OwnedPtr` from a raw pointer and a size.
///
/// # Safety
///
/// The caller must ensure that the pointer is not null and that the
/// size is correct.
pub unsafe fn from_raw_parts_zeroed_in(ptr: NonNull<T>, size: usize, alloc: A) -> Self {
let layout = Layout::from_size_align_unchecked(size, std::mem::align_of::<T>());
alloc.alloc_zeroed(layout).expect("allocation failed");
Self { ptr, size, alloc }
}
/// Create a new `OwnedPtr` containing the given value.
pub fn new(x: T) -> Self
where
A: Default,
{
Self::new_in(x, A::default())
}
/// Create a new `OwnedPtr` containing the given value.
pub fn new_in(x: T, alloc: A) -> Self {
let boxed = Box::new(x);
let ptr = Box::into_raw(boxed);
Self {
ptr: NonNull::new(ptr).expect("can never be NULL as it is a pointer to a valid T"),
size: std::mem::size_of::<T>(),
alloc,
}
}
/// Create a new `OwnedPtr` containing the given value.
pub fn from_vec(mut x: Vec<T>) -> Self {
let ptr = x.as_mut_ptr();
let size = x.len() * std::mem::size_of::<T>();
std::mem::forget(x);
Self {
ptr: NonNull::new(ptr).expect("can never be NULL as it is a pointer to a valid Vec<T>"),
size,
alloc: System,
}
}
/// Create a new `OwnedPtr` containing the given value.
pub fn from_slice(x: &[T]) -> Self
where
T: Clone,
{
let x = x.to_vec();
Self::from_vec(x)
}
}
impl OwnedPtr<u8, System> {
/// Reallocate the memory in this `OwnedPtr` with a new allocator.
pub fn reallocate<R: Allocator + Default>(self) -> Result<OwnedPtr<u8, R>, AllocError> {
let alloc = R::default();
let layout = Layout::from_size_align(self.size, std::mem::align_of::<u8>())
.expect("`Layout` can always be created with align_of created align");
let ptr = alloc.allocate(layout)?;
let src = self.ptr.as_ptr() as *const u8;
let dest = ptr.as_ptr() as *mut u8;
unsafe {
std::ptr::copy_nonoverlapping(src, dest, self.size);
};
Ok(OwnedPtr {
ptr: ptr.cast(),
size: self.size,
alloc,
})
}
}
impl<T, A> Deref for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
type Target = T;
fn deref(&self) -> &Self::Target {
self.as_ref()
}
}
impl<T, A> DerefMut for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn deref_mut(&mut self) -> &mut Self::Target {
self.as_mut()
}
}
impl<T, A> AsRef<T> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn as_ref(&self) -> &T {
unsafe { self.ptr.as_ref() }
}
}
impl<T, A> AsMut<T> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn as_mut(&mut self) -> &mut T {
unsafe { self.ptr.as_mut() }
}
}
impl<T, A> Drop for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn drop(&mut self) {
let layout = Layout::from_size_align(self.size, std::mem::align_of::<T>()).unwrap();
self.alloc.deallocate(self.ptr.cast(), layout);
}
}
impl<T, A> From<RawPtr<T>> for OwnedPtr<T, A>
where
A: Allocator
334パターン、3.3MBのデータセットでFine-tuningしたモデル
対象のコードが16,000文字以下を対象にしたデータモデル。
use crate::alloc::{AllocError, Allocator};
use crate::ptr::RawPtr;
use std::alloc::{Layout, System};
use std::ops::{Deref, DerefMut};
use std::ptr::NonNull;
/// A pointer type for a value allocated in memory.
/// This is a type used in the `ptr` module that represents a pointer to a value
/// that has been allocated in memory. It is useful for scenarios where you want
/// to own a value and ensure it is properly deallocated, but it is not an
/// `Rc`-like shared ownership pointer.
///
/// This type is useful for cases where you want to pass ownership of a value
/// across an FFI boundary, or in cases where you want to ensure memory safety
/// of a value and the value is not safe to send across threads.
///
/// # Examples
///
/// ```
/// # use std::ptr::NonNull;
/// # use std::mem::size_of;
/// # use std::alloc::Layout;
/// # use std::alloc::System;
/// # use std::ptr::RawPtr;
/// # use std::ptr::OwnedPtr;
/// # use std::alloc::Global;
/// # use std::alloc::AllocInit;
/// # use std::alloc::AllocRef;
/// # use std::alloc::AllocError;
/// # use std::alloc::Layout;
/// # use std::alloc::AllocRef;
/// # use std::alloc::Global;
/// # use std::alloc::AllocInit;
/// # use std::ptr::OwnedPtr;
/// // Allocate memory for a single `u8`.
/// let layout = Layout::from_size_align(size_of::<u8>(), 1).unwrap();
/// let mut allocator = Global::default();
/// let ptr = allocator.allocate(layout, AllocInit::Uninitialized)?;
/// let mut ptr = RawPtr::<u8>::from(ptr);
///
/// // Write a value to the memory.
/// unsafe {
/// *ptr.as_mut() = 42;
/// }
///
/// // Read the value from the memory.
/// let value = unsafe { *ptr.as_ref() };
/// assert_eq!(value, 42);
///
/// // Free the memory.
/// unsafe {
/// ptr.deallocate();
/// }
/// # Ok::<(), AllocError>(())
/// ```
pub struct OwnedPtr<T, A = System>
where
T: Sized,
A: Allocator,
{
/// The raw pointer.
pub(in crate::ptr) ptr: NonNull<T>,
/// The size of the allocation.
pub(in crate::ptr) size: usize,
/// The allocator used to allocate the memory.
alloc: A,
}
impl<T, A> OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
/// Returns a slice to the memory.
///
/// # Safety
///
/// The caller must ensure that the size field of the `OwnedPtr` is valid.
pub unsafe fn as_slice(&self) -> &[T] {
assert_eq!(
self.size % std::mem::size_of::<T>(),
0,
"Invalid size field"
);
let ptr = self.ptr.as_ptr() as *const T;
let len = self.size / std::mem::size_of::<T>();
std::slice::from_raw_parts(ptr, len)
}
/// Returns a mutable slice to the memory.
///
/// # Safety
///
/// The caller must ensure that the size field of the `OwnedPtr` is valid.
pub unsafe fn as_slice_mut(&mut self) -> &mut [T] {
assert_eq!(
self.size % std::mem::size_of::<T>(),
0,
"Invalid size field"
);
let ptr = self.ptr.as_ptr();
let len = self.size / std::mem::size_of::<T>();
std::slice::from_raw_parts_mut(ptr, len)
}
/// Consumes and leaks the value, returning the raw pointer.
pub fn into_raw(self) -> RawPtr<T> {
let ptr = self.ptr;
let size = self.size;
std::mem::forget(self);
RawPtr { ptr, size }
}
/// Consumes and leaks the slice, returning the raw pointer.
///
/// # Safety
///
/// The caller must ensure that the size field of the `OwnedPtr` is valid.
pub fn into_raw_slice(self) -> RawPtr<[T]> {
let ptr = self.ptr.as_ptr();
let size = self.size;
std::mem::forget(self);
unsafe { RawPtr::from_raw_parts(ptr, size) }
}
}
impl<T> OwnedPtr<T, System>
where
T: Sized,
{
/// Creates a new `OwnedPtr<T>`.
///
/// # Example
///
/// ```
/// # use std::ptr::OwnedPtr;
/// let ptr = OwnedPtr::new(42);
/// assert_eq!(*ptr, 42);
/// ```
pub fn new(x: T) -> Self {
let boxed = Box::new(x);
let ptr = Box::into_raw(boxed);
Self {
// Safety: `ptr` can never be NULL as it is a pointer to a valid `T`.
ptr: NonNull::new(ptr).expect("can never be NULL as it is a pointer to a valid T"),
size: std::mem::size_of::<T>(),
alloc: System,
}
}
/// Creates a new `OwnedPtr<T>` from a `Vec<T>`.
///
/// # Example
///
/// ```
/// # use std::ptr::OwnedPtr;
/// let ptr = OwnedPtr::from_vec(vec![1, 2, 3]);
/// assert_eq!(ptr.as_slice(), &[1, 2, 3]);
/// ```
pub fn from_vec(mut x: Vec<T>) -> Self {
let ptr = x.as_mut_ptr();
let size = x.len() * std::mem::size_of::<T>();
std::mem::forget(x);
Self {
// Safety: `ptr` can never be NULL as it is a pointer to a valid `Vec<T>`.
ptr: NonNull::new(ptr).expect("can never be NULL as it is a pointer to a valid Vec<T>"),
size,
alloc: System,
}
}
/// Creates a new `OwnedPtr<T>` from a `&[T]`.
///
/// # Example
///
/// ```
/// # use std::ptr::OwnedPtr;
/// let ptr = OwnedPtr::from_slice(&[1, 2, 3]);
/// assert_eq!(ptr.as_slice(), &[1, 2, 3]);
/// ```
pub fn from_slice(x: &[T]) -> Self
where
T: Clone,
{
let x = x.to_vec();
Self::from_vec(x)
}
}
impl OwnedPtr<u8, System> {
/// Reallocate the memory using the given allocator.
///
/// # Example
///
/// ```
/// # use std::ptr::OwnedPtr;
/// # use std::alloc::System;
/// let ptr = OwnedPtr::<u8>::new(42);
/// let ptr = ptr.reallocate::<System>().unwrap();
/// assert_eq!(*ptr, 42);
/// ```
pub fn reallocate<R: Allocator + Default>(self) -> Result<OwnedPtr<u8, R>, AllocError> {
let alloc = R::default();
let layout =
Layout::from_size_align(self.size, std::mem::align_of::<u8>())
.expect("`Layout` can always be created with align_of created align");
let ptr = alloc.allocate(layout)?;
let src = self.ptr.as_ptr() as *const u8;
let dest = ptr.as_ptr() as *mut u8;
unsafe {
std::ptr::copy_nonoverlapping(src, dest, self.size);
};
Ok(OwnedPtr {
ptr: ptr.cast(),
size: self.size,
alloc,
})
}
}
impl<T, A> Deref for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
type Target = T;
fn deref(&self) -> &Self::Target {
self.as_ref()
}
}
impl<T, A> DerefMut for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn deref_mut(&mut self) -> &mut Self::Target {
self.as_mut()
}
}
impl<T, A> AsRef<T> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn as_ref(&self) -> &T {
// Safety: `ptr` can never be NULL as it is a pointer to a valid `T`.
unsafe { self.ptr.as_ref() }
}
}
impl<T, A> AsMut<T> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn as_mut(&mut self) -> &mut T {
// Safety: `ptr` can never be NULL as it is a pointer to a valid `T`.
unsafe { self.ptr.as_mut() }
}
}
impl<T, A> Drop for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn drop(&mut self) {
let layout =
Layout::from_size_align(self.size, std::mem::align_of
410パターン、5.1MBのデータセットでFine-tuningしたモデル
対象のコードが32,000文字以下を対象にしたデータモデル。
//! This module provides a pointer type, `OwnedPtr`, which
//! owns the memory it points to and automatically frees it
//! when it goes out of scope. It is the owned counterpart
//! to [`RawPtr`].
//!
//! # Example
//!
//! ```
//! use std::ptr::NonNull;
//! use owned_ptr::{OwnedPtr, RawPtr};
//!
//! let x = 5;
//! let ptr = OwnedPtr::new(x);
//! assert_eq!(ptr.as_ref(), &x);
//! let raw: RawPtr<_> = ptr.into();
//! assert_eq!(unsafe { raw.as_ref() }, &x);
//! ```
//!
//! [`RawPtr`]: crate::ptr::RawPtr
use crate::alloc::{AllocError, Allocator};
use crate::ptr::RawPtr;
use std::alloc::{Layout, System};
use std::ops::{Deref, DerefMut};
use std::ptr::NonNull;
/// A pointer type which owns the memory it points to.
pub struct OwnedPtr<T, A = System>
where
T: Sized,
A: Allocator,
{
/// The raw pointer.
pub(in crate::ptr) ptr: NonNull<T>,
/// The size of the allocation.
pub(in crate::ptr) size: usize,
alloc: A,
}
impl<T, A> OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
/// Constructs a new `OwnedPtr<T>` from a `T`.
///
/// # Panics
///
/// Panics if the allocation fails.
pub fn new(x: T) -> Self {
let boxed = Box::new(x);
let ptr = Box::into_raw(boxed);
Self {
ptr: NonNull::new(ptr).expect("can never be NULL as it is a pointer to a valid T"),
size: std::mem::size_of::<T>(),
alloc: System,
}
}
/// Constructs a new `OwnedPtr<T>` from a `Vec<T>`.
///
/// # Panics
///
/// Panics if the allocation fails.
pub fn from_vec(mut x: Vec<T>) -> Self {
let ptr = x.as_mut_ptr();
let size = x.len() * std::mem::size_of::<T>();
std::mem::forget(x);
Self {
ptr: NonNull::new(ptr).expect("can never be NULL as it is a pointer to a valid Vec<T>"),
size,
alloc: System,
}
}
/// Constructs a new `OwnedPtr<T>` from a `&[T]`.
///
/// # Panics
///
/// Panics if the allocation fails.
pub fn from_slice(x: &[T]) -> Self
where
T: Clone,
{
let x = x.to_vec();
Self::from_vec(x)
}
}
impl OwnedPtr<u8, System> {
/// Reallocate the memory pointed to by `self` using a different allocator.
///
/// # Errors
///
/// Returns an error if the reallocation fails.
pub fn reallocate<R: Allocator + Default>(self) -> Result<OwnedPtr<u8, R>, AllocError> {
let alloc = R::default();
let layout = Layout::from_size_align(self.size, std::mem::align_of::<u8>())
.expect("`Layout` can always be created with align_of created align");
let ptr = alloc.allocate(layout)?;
let src = self.ptr.as_ptr() as *const u8;
let dest = ptr.as_ptr() as *mut u8;
// SAFETY: `src` and `dest` are guaranteed to be non-null because `self.ptr` is non-null.
// `self.size` is guaranteed to be non-zero because it is the size of `u8`.
unsafe {
std::ptr::copy_nonoverlapping(src, dest, self.size);
};
Ok(OwnedPtr {
ptr: ptr.cast(),
size: self.size,
alloc,
})
}
}
impl<T, A> Deref for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
type Target = T;
fn deref(&self) -> &Self::Target {
self.as_ref()
}
}
impl<T, A> DerefMut for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn deref_mut(&mut self) -> &mut Self::Target {
self.as_mut()
}
}
impl<T, A> AsRef<T> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn as_ref(&self) -> &T {
// SAFETY: This is safe because the `OwnedPtr` owns the memory it points to and
// is responsible for ensuring that it is always valid.
unsafe { self.ptr.as_ref() }
}
}
impl<T, A> AsMut<T> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn as_mut(&mut self) -> &mut T {
// SAFETY: This is safe because the `OwnedPtr` owns the memory it points to and
// is responsible for ensuring that it is always valid.
unsafe { self.ptr.as_mut() }
}
}
impl<T, A> Drop for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn drop(&mut self) {
let layout = Layout::from_size_align(self.size, std::mem::align_of::<T>()).unwrap();
// SAFETY: This is safe because the `OwnedPtr` owns the memory it points to and
// is responsible for ensuring that it is always valid.
self.alloc.deallocate(self.ptr.cast(), layout);
}
}
impl<T, A> From<RawPtr<T>> for OwnedPtr<T, A>
where
A: Allocator + Default,
{
fn from(value: RawPtr<T>) -> Self {
Self {
ptr: value.ptr,
size: value.size,
alloc: A::default(),
}
}
}
impl<T, A> From<RawPtr<[T]>> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator + Default,
{
fn from(value: RawPtr<[T]>) -> Self {
Self {
ptr: value.ptr.cast(),
size: value.size,
alloc: A::default(),
}
}
}
impl<T, A> From<OwnedPtr<T, A>> for RawPtr<T>
where
A: Allocator + Default,
{
fn from(value: OwnedPtr<T, A>) -> Self {
value.into_raw()
}
}
impl<T, A> From<OwnedPtr<T, A>> for RawPtr<[T]>
where
A: Allocator + Default,
{
fn from(value: OwnedPtr<T, A>) -> Self {
value.into_raw_slice()
}
}
484パターン、12MBのデータセットでFine-tuningしたモデル
対象のコードすべてを対象にしたデータモデル。
//! This module contains the definition of the `OwnedPtr` type. `OwnedPtr` is a
//! pointer type that owns its memory, unlike `Box`. It is useful for
//! representing pointers to memory that is not managed by Rust, such as
//! memory allocated by C libraries.
//!
//! The `OwnedPtr` type is similar to `Box`, but it does not have a destructor,
//! so it can be used to represent pointers to memory that should not be freed
//! when the pointer is dropped. It also has a `size` field, which makes it
//! possible to use `OwnedPtr` with unsized types.
//!
//! `OwnedPtr` can be used to represent both owned pointers and slices. When
//! used to represent a pointer to a single value, it is similar to `Box`. When
//! used to represent a slice, it is similar to `Vec`.
//!
//! `OwnedPtr` is implemented in terms of `RawPtr`, which is a pointer type
//! that does not own its memory. The `from_raw` and `into_raw` methods can be
//! used to convert between `OwnedPtr` and `RawPtr`.
//!
//! `OwnedPtr` is generic over an allocator type. The allocator type must
//! implement the `Allocator` trait. The `System` allocator is used by default.
//! To use a different allocator, create an `OwnedPtr` type alias that specifies
//! the allocator type.
//!
//! The `OwnedPtr` type is defined in the `ptr` module, so that it can be
//! re-exported from the `alloc` module.
use crate::alloc::{AllocError, Allocator};
use crate::ptr::RawPtr;
use std::alloc::{Layout, System};
use std::ops::{Deref, DerefMut};
use std::ptr::NonNull;
/// A pointer type that owns its memory.
///
/// `OwnedPtr` is similar to `Box`, but it does not have a destructor, so it can
/// be used to represent pointers to memory that should not be freed when the
/// pointer is dropped. It also has a `size` field, which makes it possible to
/// use `OwnedPtr` with unsized types.
///
/// `OwnedPtr` can be used to represent both owned pointers and slices. When
/// used to represent a pointer to a single value, it is similar to `Box`. When
/// used to represent a slice, it is similar to `Vec`.
///
/// `OwnedPtr` is generic over an allocator type. The allocator type must
/// implement the `Allocator` trait. The `System` allocator is used by default.
/// To use a different allocator, create an `OwnedPtr` type alias that specifies
/// the allocator type.
///
/// The `OwnedPtr` type is defined in the `ptr` module, so that it can be
/// re-exported from the `alloc` module.
pub struct OwnedPtr<T, A = System>
where
T: Sized,
A: Allocator,
{
/// The pointer.
///
/// This field is `pub(in crate::ptr)` so that `OwnedPtr` can be used by
/// other types in the `ptr` module.
pub(in crate::ptr) ptr: NonNull<T>,
/// The size of the memory block, in bytes.
///
/// This field is `pub(in crate::ptr)` so that `OwnedPtr` can be used by
/// other types in the `ptr` module.
pub(in crate::ptr) size: usize,
// The allocator.
alloc: A,
}
impl<T, A> OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
/// Constructs an `OwnedPtr` from a pointer and size.
///
/// # Safety
///
/// The pointer must be a valid pointer to memory that was allocated by
/// the allocator, and the size must be the size of the memory block, in
/// bytes.
pub unsafe fn from_raw_parts(ptr: *mut T, size: usize, alloc: A) -> Self {
Self {
ptr: NonNull::new_unchecked(ptr),
size,
alloc,
}
}
/// Returns a pointer to the memory block.
pub fn as_ptr(&self) -> *mut T {
self.ptr.as_ptr()
}
/// Returns the size of the memory block, in bytes.
pub fn size(&self) -> usize {
self.size
}
/// Returns a reference to the allocator.
pub fn allocator(&self) -> &A {
&self.alloc
}
/// Returns a mutable reference to the allocator.
pub fn allocator_mut(&mut self) -> &mut A {
&mut self.alloc
}
/// Allocates a memory block for a value.
///
/// This method is similar to `Box::new`, but it uses the specified
/// allocator instead of the global allocator.
pub fn new(x: T) -> Self {
let boxed = Box::new(x);
let ptr = Box::into_raw(boxed);
Self {
// SAFETY: `ptr` can never be NULL as it is a pointer to a valid `T`.
ptr: NonNull::new(unsafe { ptr }).expect("can never be NULL as it is a pointer to a valid T"),
size: std::mem::size_of::<T>(),
alloc: A::default(),
}
}
/// Allocates a memory block for a slice.
///
/// This method is similar to `Vec::from` or `Vec::from_raw_parts`, but it
/// uses the specified allocator instead of the global allocator.
pub fn from_vec(mut x: Vec<T>) -> Self {
let ptr = x.as_mut_ptr();
let size = x.len() * std::mem::size_of::<T>();
std::mem::forget(x);
Self {
// SAFETY: `ptr` can never be NULL as it is a pointer to a valid `Vec<T>`.
ptr: NonNull::new(unsafe { ptr }).expect("can never be NULL as it is a pointer to a valid Vec<T>"),
size,
alloc: A::default(),
}
}
/// Allocates a memory block for a slice.
///
/// This method is similar to `Vec::from` or `Vec::from_raw_parts`, but it
/// uses the specified allocator instead of the global allocator.
pub fn from_slice(x: &[T]) -> Self
where
T: Clone,
{
let x = x.to_vec();
Self::from_vec(x)
}
}
impl OwnedPtr<u8, System> {
/// Reallocates the memory block.
///
/// This method is similar to `std::alloc::realloc`, but it uses the
/// specified allocator instead of the global allocator.
pub fn reallocate<R: Allocator + Default>(self) -> Result<OwnedPtr<u8, R>, AllocError> {
let alloc = R::default();
let layout = Layout::from_size_align(self.size, std::mem::align_of::<u8>())
.expect("`Layout` can always be created with align_of created align");
let ptr = alloc.allocate(layout)?;
let src = self.ptr.as_ptr() as *const u8;
let dest = ptr.as_ptr() as *mut u8;
unsafe {
std::ptr::copy_nonoverlapping(src, dest, self.size);
};
Ok(OwnedPtr {
ptr: ptr.cast(),
size: self.size,
alloc,
})
}
}
impl<T, A> Deref for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
type Target = T;
fn deref(&self) -> &Self::Target {
self.as_ref()
}
}
impl<T, A> DerefMut for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn deref_mut(&mut self) -> &mut Self::Target {
self.as_mut()
}
}
impl<T, A> AsRef<T> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn as_ref(&self) -> &T {
// SAFETY: `ptr` can never be NULL as it is a pointer to a valid `T`.
unsafe { self.ptr.as_ref() }
}
}
impl<T, A> AsMut<T> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn as_mut(&mut self) -> &mut T {
// SAFETY: `ptr` can never be NULL as it is a pointer to a valid `T`.
unsafe { self.ptr.as_mut() }
}
}
impl<T, A> Drop for OwnedPtr<T, A>
where
T: Sized,
A: Allocator,
{
fn drop(&mut self) {
let layout = Layout::from_size_align(self.size, std::mem::align_of::<T>()).unwrap();
self.alloc.deallocate(self.ptr.cast(), layout);
}
}
impl<T, A> From<RawPtr<T>> for OwnedPtr<T, A>
where
A: Allocator + Default,
{
fn from(value: RawPtr<T>) -> Self {
Self {
ptr: value.ptr,
size: value.size,
alloc: A::default(),
}
}
}
impl<T, A> From<RawPtr<[T]>> for OwnedPtr<T, A>
where
T: Sized,
A: Allocator +
おわりに
今回はTemperatureとTop Pが有効な状態で出てきた一発目を利用していたため、あまり良い比較ではありません。（本当は0にしたかったけどgpt3.5系で結果が壊れるの泣く泣く有効にした）
それでも、データセットのデータ量は正義ということで、484パターン、12MBのデータセットでFine-tuningしたモデルはなかなか良い感じのコメントになっていますね。
意外だったのはgpt-3.5、gpt-4でもそれなりにRust-Likeなコメントになっていて今までやっていたのも悪くはなかったんだなと再認識しました。
余談ですが、今回の検証で約$100ほど飛びました。
Fine-tuningしかしてませんでしたが、意外にFine-tuningが良い金額・・・。
Discussion